Hi Jasper, for the 560 response code, you should be able to find an exception in the Mendix Runtime log on your servers. I suspect it simply talks about that URL being invalid for a Mendix application.
For the performance testing, I wonder if having the proper SSO authentication is important enough to get working with the performance test. It depends on where you see the highest risk. Classically, the main risks in a custom application like this lie in the interaction between Client, Runtime and Database, e.g. queries, data volumes, concurrency, perhaps integrations. If possible I would consider adding a second login method to the performance testing environment with username/password or similar authentication and of course only have them in the test environment. If you're able to record the actions the Native Mobile Client sends to the Runtime after authentication, then you're usually all set for testing all that happens after authentication.
If you however have done a lot of customisations for the authentication or see some risk there, I would consider designing a specific performance test for that part by itself, rather than combining it with the performance test for the actual application performance. I don't know what that custom URL does, but probably someone here knows more. Can you trigger the same authentication sequence from a desktop browser where you might have easier insight into the details?
Hope it helps!