Well, you are doing it correctly. You will be required to add and event handler on the <a>
tag you have defined. I tried it using jQuery dialog.
Apex Page:
<apex:page controller="ErrorMSG" >
<apex:pageMessages escape="false" rendered="true" ></apex:pageMessages>
<link rel="stylesheet" href="//code.jquery.com/ui/1.11.4/themes/smoothness/jquery-ui.css"/>
<script src="//code.jquery.com/jquery-1.10.2.js"></script>
<script src="//code.jquery.com/ui/1.11.4/jquery-ui.js"></script>
<link rel="stylesheet" href="/resources/demos/style.css"/>
<div id="dialog" title="Basic dialog">
<p>This is the default dialog which is useful for displaying information. The dialog window can be moved, resized and closed with the 'x' icon.</p>
</div>
<script>
$j = jQuery.noConflict();
$j( "#popup" ).click(function(){
$j( "#dialog" ).dialog();
})
</script>
</apex:page>
Controller
public class ErrorMSG {
public ErrorMSG(){
ApexPages.addMessage(new ApexPages.Message(ApexPages.Severity.WARNING , 'You have received an error <b><u> <span id="popup">Click for Details</span><u></b>'));
}
}
Result

You will be required to refine the code and surround the dialog by outputpanel and re-render it if there is a failure and hide dialog at start.