First, make sure that SAML redirects to the same url as the url where the app started. If you start the app using a custom url and SAML returns with a .mendixcloud.com url, then the InAppBrowser will not close.
Second, make sure you have a recent SAML20 module and in the runtime configuration enable the checkbox "Enable mobile authentication data". If you use the Dutch language, note that the checkbox has a wrong label: it is then the second checkbox named "Enable delegated authentication". If this is not checked (or you have an older SAML20 module) then the hybrid app will not receive a token and the InAppBrowser may not close because of a bug in the JavaScript.
Third, the abovementioned bug was recently fixed by Mendix by updating the JavaScript code mentioned here: https://docs.mendix.com/howto/mobile/implement-sso-on-a-hybrid-app-with-mendix-and-saml. I advise you to revisit that page and use the new JavaScript code.
Does this help?