Our partner cape-group has made an interface to an external reporting tool (Jasper reports):
https://appstore.mendix.com/link/app/Advanced%20Reporting%20Suite
We did something similar with Microsoft SQL reporting services. We created the reports and then hosted them on our server. We then used the IFrame widget to view the reports: https://appstore.mendix.com/index3.html#1309707851692_497
You could do something similar using a BIRT server to render and run the reports, but use the iframe to view the report using a link to the report.
In reply to the additional questions:
Extra libraries go in the userlib folder. Additional deployment size will be the size of the libraries (roughly)
We faced the same problem. We built this ARS-tool to be able to make various reports from Mendix. In particular for html, pdf, docx or xls and many other formats. We put several functionalities in: like zooming, deeplinking and subreports. We chose Jasper iReports because it is opensource. If you like you can try it yourself or we can demo this solution. Just contact me.
Where would the external jar files be included? In the resources folder? What is the impact of this on deploying the project (i.e. is this 100MB+ deployed every time)?