Start by creating a VF page that includes the dialog, and uses the standard controller so you can drop it into the object's whose details your trying to add it to the page layout, since you didn't specify, I'll just use 'Account'
You need to uploaded your jQuery or whatever JS libraries your using to generate the dialog into a static resource file (as you can't remotely use a CDN on the standard details page)
<apex:page standardController="account">
<apex:includeScript value="{!URLFOR($Resource.jquery)}" />
<script type="text/javascript">
$(document).ready(function(){
$('#open-notes-dialog).click(function(){
alert('hello world');
});
});
</script>
<button id="open-notes-dialog">Add notes</button>
Add the page to the layout of your details page. Unfortunately, its kind of hard to get it to display in a correct size.
However, you will now have this VF page in an iFrame and with some creative JS can go outside of your iframe (into the parent) and rename the 'Notes & Attachments' heading, and add your dialog!
Update -- looks like you can't access static resources from the standard details page- so maybe add the javascript you need to your page statically but adding a dynamic library might be limited.
