I don't have a Solaris environment handy to test this in, but it appears that the Java runtime expects some form of X interaction. Try adding the following flag to the server invocation,
-Djava.awt.headless=true
If this doesn't work, please provide the JVM version, Solaris version and system architecture (SPARC or Intel, I assume).
Michiel, Thank you very much. Mendix platform can generate thumbnails now.