I have some experience with it if the Mendix server is also hosted on premise or in the SAP Cloud.
If it is hosted in the Mendix Cloud then best would be to set up a VPN from the Mendix Cloud to the on premise system, Mendix can help you with this. The easier and in my experience best, but perhaps more expensive, option would be to connect to SAP via the SAP Cloud Connector. You then need to host your Mendix app in the SAP Cloud and make the on premise system available in the SAP Cloud via the Cloud Connector. A very great advantage of this is that if you need to connect to any other on premise system, you can use the SAP Cloud Connector for this.
As for OData, a lot of it is automated from the Mendix side, but indeed an OData service is needed. We had the same struggles before, you need somebody who is able to build the OData service in the SAP Gateway of the ERP system. This involves some SAP and ABAP knowledge and is not plug and play like most of the Mendix development is..
Good luck!
If you have SAP onpremise, than no doubt (or rather, I hope) that you have some SAP consultants present. They can tell you all about SAP’s API’s. Because there are a lot of API’s available. For out-of-the-box API’s visit https://api.sap.com/. Don’t get overwhelmed but just type the entity you need in the searchbox. For instance ‘invoice’:
It will give you a list of available OData, REST and SOAP api’s.
Mind you, these API’s are provided by your SAP-application and of course differ per SAP version. If you are still on SAP R/3, expect more manual work for getting them up-and-running. If you are on S/4 HANA, you are more likely to get a kickstart.