PS OldGen causing java out of memory

5
Today our application was running ok after a fresh restart, at the end of the day suddenly the PS OldGen started to fill up to its max, causing the application to become inaccessible. After waiting for some minutes OldGen lowered and the application became accessible again. I checked whether any scheduled events were running, but they were not (or they were, but finished in seconds). In the m2ee log I find a lot of unloading and loading messages like: [Unloading class sun.reflect.GeneratedConstructorAccessor289]. Followed by: Exception in thread "Thread-49" java.lang.OutOfMemoryError: Java heap space at java.lang.StringCoding$StringDecoder.decode(StringCoding.java:133) at java.lang.StringCoding.decode(StringCoding.java:173) at java.lang.String.<init>(String.java:443) at java.lang.String.<init>(String.java:515) at com.microsoft.sqlserver.jdbc.DDC.convertStreamToObject(DDC.java:438) at com.microsoft.sqlserver.jdbc.ServerDTVImpl.getValue(dtv.java:2503) at com.microsoft.sqlserver.jdbc.DTV.getValue(dtv.java:193) After waiting for some minutes all worked fine again. At this moment I see that the OldGen is still rather high compared to the rest of the day (but application is working fine now. No other specifics can be found in the logs. What would be the best way to 'debug' this issue? How can I find out more about which processes might be causing this issue? Thanks! NOTE: our application is running on the current version without any new model deployments or server console updates for approx 3 Months now, the described issues started during the last 2 weeks . RESOLVED: 1 of our java processes that checks emails caused the out of memory errors. 1 user suddenly received 50000 emails in 2 days which caused a query to run extremely long. Adding a java parameter that automatically generates a heap dump in case of an out of memory, then followed by a Java heap dump analysis showed the java class causing the issue.
asked
1 answers
7

For future reference, and since this might be interesting to other readers:

We use the Eclipse Memory Analyzer (MAT) to look at heap dump files (hprof format). These can be generated automatically using JVM command line options (-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath="C:\path\to\file.hprof") or manually using monitoring tools such as JConsole or JVisualVM.

MAT has some interesting automated analysis reports, the most useful of which is "leak suspects", which will look at which threads or objects keep the most memory active. In this (relatively simple) case, there was a single thread that kept 90% of memory (>3GB) in use.

MAT also allows you to look at the current state (stack) of all threads at the time of the dump. Looking at the reported thread showed that it was executing a custom java action, which did an XPath retrieve, which ultimately hit the database. By inspecting the local variables at each stack frame, we noticed that near the top of the stack a SQL resultset was being built which used all the memory.

We sent the action and XPath/SQL query that caused this to Brian, which allowed him to identify the cause of the issue.

answered