The forgot password module uses the getApplicationRootUrl java action, this action is not checking the domain as it is in the browser bar, but on the server side. You can however overwrite it in the custom runtime settings of your environment setup.
More about Custom Runtime Settings: https://docs.mendix.com/refguide6/custom-settings#2-general-settings
Another forum post about getApplicationRootUrl not working for custom domains, with some work-a-rounds: https://forum.mendix.com/link/questions/88684.
Hope this helps.