You can indeed configure these options in the 'Extra JVM parameters' field in the Server tab of the Project Settings. It should be a single line containing all the options, e.g.
-Dfile.encoding=UTF-8 -XX:MaxPermSize=64M
The -Xmx
option (maximum Java heap size) can be configured separately in the project settings by setting 'Java heap' to 'Custom' and using the 'Maximum size (in MB)' field.