Can you reproduce this consistently? If so, you could monitor your network traffic to see where the request actually gets stuck.
If your application is deployed behind a web server (as recommended), make sure to proxy '/file' requests to the Mendix runtime as well. In such a case you might also have a size limit on these requests set in your web server that could be lower than the 5 MiB in your model (but I think this would result in a different error message).