Thanks to Mx Support who helped us out. In our case it was related to widgets from the appstore that were not yet compatible with Mx5 (tooltips to be precise). Removing those incompatible widgets from the page resolved the problem.
So I would recommend to check the pages where you have issues, and see if you can identify a widget that may cause this unwanted behavior.