As far as I know, configuring connection security settings such as SSL and client certificates must be done at the web server that you use in front of the Mendix Runtime: IIS, Nginx, Jetty, etc. Searching Google for "configure client certificates [webserver]" should point you in the right direction.
Edit: Mendix allows plugging in a custom login action in Java to perform authentication, that's how the LDAP integration works for example. I have no idea how you would connect this to client authentication of a web server though.
You can register your own requesthandler via the Core API. If you allow anonymous users (that aren't allowed to do anything), you could setup the custom requesthandler to verify the request and login the user.
As Benny noted, this is how the LDAP appstore thingy works.
I have found a potential solution that takes the user through a longer route to get into the app. This solution involves IDP and SAML and its like this: User come to my app and accept terms on home page → Click on login button that will take them to IDP (okta) → IDP will do authentication using CAC/PIV → Send token back to me in SAML → Assuming everything is valid, the user gets logged in.
I’d prefer to keep the users in my own app and allow them to do authentication using x509 in my app.
Any help is much appreciated…
P.S. I am trying to revive this very old post hence adding an issue/question in the answer section.