Mendix can able to code in java and using lib .jar.
In case of using dll, there is a work around to write java action to call dll. There are some tool to convert dll to jar, but not 100 % perfect, anyway you still need to re-write to jar.
As Quang says, the Mendix runtime is Java.
I don’t remember ever seeing anyone getting a Java to DLL action working on Mendix because of the security restrictions, but I could be wrong.
It would be easier to look to provide a web service wrapper around the DLL and then use Mendix’s “Call REST Service” or “Call Web Service” actions to access the functionality.
Hope this helps.