I am no specialist in this matter but may be this blog post give you some clue on how to set up the request handler: https://www.mendix.com/blog/requesthandlers-at-your-service/
Regards,
Ronald
You're most likely missing the rewrite rule for the /rest/ endpoint. That's what tells IIS to forward a request through from itself to the Mendix runtime. See this documentation for how exactly to set one up: