That is only posible in a newer Mendix version. In 8.8.2, this feature does not exist. You can only “open” the page that is already open in the background. That can be done in a microflow. You need to know on which page you started. If your popups are used on multiple pages, you need some sort of helper.
From Mendix Studio Pro v8.14 and onwards you can choose to close multiple pages.