This is expected behavior. As long as the type of the return-variable is a boolean, then return value 'void’ will become false. Now you have set the type of the return-variable to integer, Java is asked to convert the value 'void’ to a long, and that is not possible.
Just leave the return type on boolean and do the transformation from boolean to integer in your microflow after calling MQTT-activity.