Hi Jammula,
You can go to the properties of the button and select the onclick as "Close page". I think this should work fine because the actual OnClick - directing to a link is handled by your JS.
PS: I would recommend you to first research on ways to implement the same functionality within Mendix first before of opting for JS code manipulation. Good Luck!
EDIT: Please try the below steps and let us know if this what is expected.
1. In your page add a Action Button with 'OnClick' value as "Close page" and 'Class' as "TestButton"
2. Add a HTML snippet (bottom of the page preferably) if not done already.
3. Add the below JS code replacing the link with your URL.
$('.TestButton').bind('click', function()
{
window.open('http://www.mendix.com','_blank','resizable=yes');
});
3. Modify the window.open params based on your requirements, to open URL in the same tab or a new tab etc. etc.
Thanks,
AD