After some trial and error attempts we figured that the server was not hosting the complete chain which caused this issue.
One other intresting thing we noticed is that when using self signed client certificates, it seems that you need to have this certificate both as ca and as client certificate because otherwise Mendix won't send the client certificate to the server resulting in an error.