When you select custom authentication in your service, you can select the microflow you will be using and you can define the parameters that will be passed, the access token in your case.
In your microflow you need to decode and validate the JWT token and then return a System.User object if the access token is valid. If it is not valid you return an empty user entity and if I am not mistaken the service will then return a 401.