Adjusting the mx.underlay would be my last approach and you would need to get your hands on the proprietary Mendix code, which is not publicly shared.
I red something about changing a popup into an overlay, by adding a class to the second popup-page and change the styling using scss.
And you may find a solution in these previous forum-posts:
https://forum.mendix.com/link/questions/91182, making the second a modal-popup
https://forum.mendix.com/link/questions/90681, adding some smart javascript