Unfortunately for the profile you wont be able access as a lookup as we all know there is no such field in salesforce thats lookup to profile

The alternative solution could be to roll your own custom lookup window as shown in below blog

http://blog.jeffdouglas.com/2011/08/12/roll-your-own-salesforce-lookup-popup-window/

OR just think of a new window open in javascript and pass the value from child window to the parent window .

Lets say i want to open a visualforce page(PageOpener) as a pop up window through a sample vf page named WindowOpenerTest

 

Code for the WindowOpenerTest(Parent page)

<apex:page id="page">
<apex:form id="theform">
    <script type="text/javascript">
             var newWin;

 function openWin(){
    var url="/apex/PageOpener";
    newWin=window.open(url, 'Popup','height=500,width=800,left=200,top=100');
    newWin.focus();
} 

 function parentalert(camp){
     document.getElementById('{!$Component.hello}').value=camp;
     //alert(a);
//alert('In parent Window<<<<<<<'+camp);
  newWin.close();
}


 </script>
      <apex:inputText id="hello" onclick="openWin();" />
    </apex:form>
    </apex:page>

Code for the child window(PageOpener) and script for the communication from child to parent window

<apex:page sidebar="false" showHeader="false">

<apex:form >
    <!-- Begin Default Content REMOVE THIS -->
    <h1>Congratulations</h1>
            This is your new Page
            <!-- End Default Content REMOVE THIS -->

    <apex:outputLink id="theLink"
        onclick="sendPopUpValues('{!$User.Alias}')" styleClass="p"> Link
                 </apex:outputLink>

    <script type="text/javascript">
         function sendPopUpValues(camp){//sending user Alias to the parent window through script
            window.opener.parentalert(camp);
   }        
   </script>

</apex:form>
</apex:page>

The above code should get you started and help you on how to send values between parent and child windows.

Unfortunately for the profile you wont be able access as a lookup as we all know there is no such field in salesforce thats lookup to profile

The alternative solution could be to roll your own custom lookup window as shown in below blog

http://blog.jeffdouglas.com/2011/08/12/roll-your-own-salesforce-lookup-popup-window/

OR just think of a new window open in javascript and pass the value from child window to the parent window .

Lets say i want to open a visualforce page(PageOpener) as a pop up window through a sample vf page named WindowOpenerTest

Code for the WindowOpenerTest(Parent page)

<apex:page id="page">
<apex:form id="theform">
    <script type="text/javascript">
             var newWin;

 function openWin(){
    var url="/apex/PageOpener";
    newWin=window.open(url, 'Popup','height=500,width=800,left=200,top=100');
    newWin.focus();
} 

 function parentalert(camp){
     document.getElementById('{!$Component.hello}').value=camp;
     //alert(a);
//alert('In parent Window<<<<<<<'+camp);
  newWin.close();
}


 </script>
      <apex:inputText id="hello" onclick="openWin();" />
    </apex:form>
    </apex:page>

Code for the child window(PageOpener) and script for the communication from child to parent window

<apex:page sidebar="false" showHeader="false">

<apex:form >
    <!-- Begin Default Content REMOVE THIS -->
    <h1>Congratulations</h1>
            This is your new Page
            <!-- End Default Content REMOVE THIS -->

    <apex:outputLink id="theLink"
        onclick="sendPopUpValues('{!$User.Alias}')" styleClass="p"> Link
                 </apex:outputLink>

    <script type="text/javascript">
         function sendPopUpValues(camp){//sending user Alias to the parent window through script
            window.opener.parentalert(camp);
   }        
   </script>

</apex:form>
</apex:page>

The above code should get you started and help you on how to send values between parent and child windows.