A couple of things to check:
- if you remove the custom URL and then visit the home page does the infinite loop still occur?
- is the custom URL definitely going to the landing page you expect?
- if you make that landing page anonymous temporarily does the redirecting still occur?
My suspicion is that this is a role access thing as you're actually being sent to a different page than you think you are as part of the CNAME record.
But, start with those, an if not, come back and post the URL of a test environment, and a pic of the cname record etc so we can see if we can spot anything else.
(in terms of runtime settings, the only other one we have set (and we have a custom domain) is "com.mendix.core.SameSiteCookies" set to "None")