You can create a specific API user with for instance a specific API role in your app to deal with that. In the custom authentication microflow you then retrieve that user. Depending on your needs you can add custom logic and return different users depending on who is calling your API.
You can turn off authentication at the service level, then implement your custom checks in the endpoint microflows themselves. So include the HttpResponse object as a parameter in each endpoint microflow. At the beginning of each flow, perform your auth checks, and if they fail then change the response code accordingly and return.