Hi,
.rtf is different from the others because it is using ANSI character set (by default). The interface is choking on the special characters in the other file types. I didn't look at the API definition, but 99 out of 100 will support Base64Encode.
Use https://marketplace.mendix.com/link/component/170 and documentation https://docs.mendix.com/appstore/modules/community-commons-function-library/ to import that functionality. Use
Base64EncodeFile
– This converts an un-encoded file to a base 64-encoded string.
and throw the output of that function to the API.
Go Make IT