When uploading/downloading a file using the standard file widget, it will be stored to, or retrieved from a FileDocument object.
You could create a bunch of custom url-handlers in Java, using their own security system, which interact with the main part of the mendix application and the external system, but that sounds like much work to get it right.
(Anyway, why the need to transport a private key? Typically private keys should be generated by/at their owner. When using PKI like a SSL CA, you only transport certificate requests and certificates, not keys. It's a basic idea behind PKI to never have to transfer secret keys.)
What webserver do you use (IIS or Apache)? You can handle that there.
Can I upload/download some file (a keypair) without storing this file in Mendix? Samba