What I usually do is create a non-persitable entity with an unlimited string attribute, and map the binary attribute to that string attribute. Mendix will then convert the MTOM attachment to a Base64 string, which you can then convert to a filedocument, using the CommunityCommons java action Base64DecodeToFile.
Also, make sure you have the MTOM option enabled in the Consumed webservice configuration.
Mendix supports MTOM/XOP, so it should be possible.
Maybe you can try not to use a FileDocument but a new Entity with it's own Binary attribute.