Yes, this is certainly possible. In the Server Settings dialog of the Modeler, you can set the maximum Java heap size for development and test modes. The property is called 'Maximum Java heap size' and is located in the 'Advanced' group box of the dialog. You can configure this property so that the Java VM uses a specified maximum heap size.
Configuring this property will add an extra option to the java command that is used to start the runtime. An example java command to start the runtime with a maximum heap size of 256MB could be:
java -Djava.library.path=WEB-INF/lib -Xmx256M -jar WEB-INF/lib/mx.jar
When starting the Mendix Runtime manually you can also specify the heapspace using the following options on the command line:
~/tmp/wikitest$ java -server -Xms128M -Xmx128M -jar WEB-INF/lib/mx.jar
Change 128 to the amount of mega bytes you want to set the heap size to.
Specifying an equal fixed minimum and maximum value for the JVM Heap size prevents the JVM from resizing its heap space and minimizes unnecessary garbage collections.
For detailed info see this manual from SUN