This forum post describes the process in two steps: https://forum.mendix.com/link/questions/104097
And see the documentation here for the one step: https://docs.mendix.com/refguide/call-rest-action#6-3-form-data
Use Request / Form data. There you can create the key and a value. For the filedocument you need to pass the Content of the filedocument in the value.
Regards,
Ronald