In Mendix Private Cloud, this is usually handled at the infrastructure level, not in Studio Pro.
For an incoming REST endpoint, Mendix does not provide an app-level setting to enforce client certificate authentication on the published service itself. The common approach is to configure mTLS on the Ingress / reverse proxy / load balancer in front of the app. Mendix Private Cloud supports exposure through options like Kubernetes Ingress, OpenShift Route, or Service Only, and Service Only gives the most control if you want to manage security yourself.
So yes, your assumption is correct: in Private Cloud, TLS certificate authentication for REST endpoints is generally implemented outside Mendix, at the platform/network layer. If you share which ingress you use, the exact setup can be more specific.