When deploying Mendix in Dockers, it is my experience that you will need to refer to the documentation frequently, both the docker-mendix-buildpack and the cf-mendix-buildpack.
The latter one contains the way to set the heap size for a Docker using an environment variable:
HEAP_SIZE 512M
[edit]
You set this environment variable the same way you set any environment variable. At my client, we have a fully customized setup, my code example won't help, but the documentation shows the following example to set environment variables when running your Docker:
docker run -it \
-e ADMIN_PASSWORD=Password1! \
-e DATABASE_ENDPOINT=postgres://username:password@host:port/mendix \
mendix/mendix-buildpack:v1.2
Adding the heap size to this, it would look like:
docker run -it \
-e ADMIN_PASSWORD=Password1! \
-e DATABASE_ENDPOINT=postgres://username:password@host:port/mendix \
-e HEAP_SIZE=1024M \
mendix/mendix-buildpack:v1.2
Finally, having looked at my code, I actually use two environment variable to control heap size: HEAP_SIZE and MEMORY_LIMIT. HEAP_SIZE is set to 90% of MEMORY_LIMIT (so, for a Docker with 1GB memory, we use MEMORY_LIMIT=1024M and HEAP_SIZE=920M). I didn't find that in the documentation, but it is in the Python code of the cf-mendix-buildpack. So, the final configuration we end up with looks like this:
docker run -it \
-e ADMIN_PASSWORD=Password1! \
-e DATABASE_URL=sqlserver://username:password@host:port/mendix \
-e MEMORY_LIMIT=1024M \
-e HEAP_SIZE=920M \
mendix/mendix-buildpack:v1.2