1

Our application crashed this morning (around 8 ) but I can't seem to figure out why. The logs do not show much, no users where logged in and there are currently no scheduled events in the application. Any suggestions on where to start looking?   Mar 16 19:04:27.068 - INFO - WebUI: Anonymous user 'Anonymous_0875ead4-8a28-4747-be8a-05417592d802' created (Number of concurrent sessions: 2). Mar 16 19:16:49.327 - INFO - Core: Removing session for user 'Anonymous_0875ead4-8a28-4747-be8a-05417592d802' Mar 17 07:57:14.607 - CRITICAL - M2EE: Java heap space Mar 17 07:57:14.607 - ERROR - M2EE: An error occurred while executing action 'get_all_thread_stack_traces'. Mar 17 07:57:20.337 - ERROR - M2EE: (1/23) com.mendix.m2ee.api.AdminException: JVM Error! Mar 17 07:57:21.747 - ERROR - M2EE: (2/23) #011at com.mendix.m2ee.appcontainer.server.handler.HttpAdminHandler.handle(HttpAdminHandler.java:149) Mar 17 07:57:21.747 - ERROR - M2EE: (3/23) Mar 17 07:57:21.747 - ERROR - M2EE: (4/23) Caused by: java.lang.OutOfMemoryError: Java heap space Mar 17 07:57:21.747 - ERROR - M2EE: (5/23) #011at java.util.Arrays.copyOf(Arrays.java:3332) Mar 17 07:57:21.747 - ERROR - M2EE: (6/23) #011at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124) Mar 17 07:57:21.747 - ERROR - M2EE: (7/23) #011at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:649) Mar 17 07:57:24.598 - ERROR - M2EE: (8/23) #011at java.lang.StringBuffer.append(StringBuffer.java:381) Mar 17 07:57:24.598 - ERROR - M2EE: (9/23) #011at org.json.JSONObject.toString(JSONObject.java:1396) Mar 17 07:57:24.598 - ERROR - M2EE: (10/23) #011at com.mendix.m2ee.appcontainer.server.handler.HttpAdminHandler.handle(HttpAdminHandler.java:127) Mar 17 07:57:24.598 - ERROR - M2EE: (11/23) #011at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) Mar 17 07:57:24.598 - ERROR - M2EE: (12/23) #011at org.eclipse.jetty.server.Server.handle(Server.java:368) Mar 17 07:57:24.598 - ERROR - M2EE: (13/23) #011at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489) Mar 17 07:57:25.984 - ERROR - M2EE: (14/23) #011at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:953) Mar 17 07:57:25.985 - ERROR - M2EE: (15/23) #011at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1014) Mar 17 07:57:25.985 - ERROR - M2EE: (16/23) #011at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:861) Mar 17 07:57:27.390 - ERROR - M2EE: (17/23) #011at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240) Mar 17 07:57:31.740 - ERROR - M2EE: (18/23) #011at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) Mar 17 07:57:31.740 - ERROR - M2EE: (19/23) #011at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:628) Mar 17 07:57:31.741 - ERROR - M2EE: (20/23) #011at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52) Mar 17 07:57:31.741 - ERROR - M2EE: (21/23) #011at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) Mar 17 07:57:31.742 - ERROR - M2EE: (22/23) #011at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) Mar 17 07:57:31.742 - ERROR - M2EE: (23/23) #011at java.lang.Thread.run(Thread.java:745) Mar 17 07:57:31.743 - WARNING - Jetty: / Mar 17 07:57:31.743 - WARNING - Jetty: (1/23) java.lang.OutOfMemoryError: Java heap space Mar 17 07:57:31.744 - WARNING - Jetty: (2/23) #011at java.util.Arrays.copyOf(Arrays.java:3332) Mar 17 07:57:31.744 - WARNING - Jetty: (3/23) #011at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124) Mar 17 07:57:31.744 - WARNING - Jetty: (4/23) #011at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448) Mar 17 07:57:31.744 - WARNING - Jetty: (5/23) #011at java.lang.StringBuffer.append(StringBuffer.java:270) Mar 17 07:57:31.744 - WARNING - Jetty: (6/23) #011at org.json.JSONObject.toString(JSONObject.java:1401) Mar 17 07:57:31.744 - WARNING - Jetty: (7/23) #011at org.json.JSONObject.valueToString(JSONObject.java:1535) Mar 17 07:57:31.744 - WARNING - Jetty: (8/23) #011at org.json.JSONObject.toString(JSONObject.java:1401) Mar 17 07:57:31.744 - WARNING - Jetty: (9/23) #011at com.mendix.m2ee.appcontainer.server.handler.HttpAdminHandler.processAdminException(HttpAdminHandler.java:251) Mar 17 07:57:31.744 - WARNING - Jetty: (10/23) #011at com.mendix.m2ee.appcontainer.server.handler.HttpAdminHandler.handle(HttpAdminHandler.java:152) Mar 17 07:57:31.744 - WARNING - Jetty: (11/23) #011at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) Mar 17 07:57:31.745 - WARNING - Jetty: (12/23) #011at org.eclipse.jetty.server.Server.handle(Server.java:368) Mar 17 07:57:31.745 - WARNING - Jetty: (13/23) #011at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489) Mar 17 07:57:33.131 - WARNING - Jetty: (14/23) #011at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:953) Mar 17 07:57:33.131 - WARNING - Jetty: (15/23) #011at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1014) Mar 17 07:57:33.132 - WARNING - Jetty: (16/23) #011at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:861) Mar 17 07:57:33.132 - WARNING - Jetty: (17/23) #011at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240) Mar 17 07:57:33.132 - WARNING - Jetty: (18/23) #011at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) Mar 17 07:57:33.134 - WARNING - Jetty: (19/23) #011at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:628) Mar 17 07:57:33.134 - WARNING - Jetty: (20/23) #011at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52) Mar 17 07:57:33.134 - WARNING - Jetty: (21/23) #011at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) Mar 17 07:57:33.134 - WARNING - Jetty: (22/23) #011at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) Mar 17 07:57:33.134 - WARNING - Jetty: (23/23) #011at java.lang.Thread.run(Thread.java:745) Mar 17 07:58:02.069 - CRITICAL - M2EE: Java heap space Mar 17 07:58:02.069 - ERROR - M2EE: An error occurred while executing action 'get_all_thread_stack_traces'. Mar 17 07:58:03.467 - ERROR - M2EE: (1/23) com.mendix.m2ee.api.AdminException: JVM Error! Mar 17 07:58:03.467 - ERROR - M2EE: (2/23) #011at com.mendix.m2ee.appcontainer.server.handler.HttpAdminHandler.handle(HttpAdminHandler.java:149) Mar 17 07:58:03.467 - ERROR - M2EE: (3/23) Mar 17 07:58:03.467 - ERROR - M2EE: (4/23) Caused by: java.lang.OutOfMemoryError: Java heap space Mar 17 07:58:03.467 - ERROR - M2EE: (5/23) #011at java.util.Arrays.copyOf(Arrays.java:3332) Mar 17 07:58:04.854 - ERROR - M2EE: (6/23) #011at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124) Mar 17 07:58:04.854 - ERROR - M2EE: (7/23) #011at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:649) Mar 17 07:58:04.854 - ERROR - M2EE: (8/23) #011at java.lang.StringBuffer.append(StringBuffer.java:381) Mar 17 07:58:04.854 - ERROR - M2EE: (9/23) #011at org.json.JSONObject.toString(JSONObject.java:1396) Mar 17 07:58:04.854 - ERROR - M2EE: (10/23) #011at com.mendix.m2ee.appcontainer.server.handler.HttpAdminHandler.handle(HttpAdminHandler.java:127) Mar 17 07:58:04.854 - ERROR - M2EE: (11/23) #011at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) Mar 17 07:58:04.854 - ERROR - M2EE: (12/23) #011at org.eclipse.jetty.server.Server.handle(Server.java:368) Mar 17 07:58:04.854 - ERROR - M2EE: (13/23) #011at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489) Mar 17 07:58:04.854 - ERROR - M2EE: (14/23) #011at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:953) Mar 17 07:58:04.854 - ERROR - M2EE: (15/23) #011at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1014) Mar 17 07:58:09.085 - ERROR - M2EE: (16/23) #011at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:861) Mar 17 07:58:09.086 - ERROR - M2EE: (17/23) #011at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240) Mar 17 07:58:09.086 - ERROR - M2EE: (18/23) #011at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) Mar 17 07:58:09.086 - ERROR - M2EE: (19/23) #011at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:628) Mar 17 07:58:11.856 - ERROR - M2EE: (20/23) #011at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52) Mar 17 07:58:11.856 - ERROR - M2EE: (21/23) #011at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) Mar 17 07:58:11.856 - ERROR - M2EE: (22/23) #011at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) Mar 17 07:58:11.856 - ERROR - M2EE: (23/23) #011at java.lang.Thread.run(Thread.java:745) Mar 17 07:58:18.772 - WARNING - Jetty: / Mar 17 07:58:18.772 - WARNING - Jetty: (1/21) java.lang.OutOfMemoryError: Java heap space Mar 17 07:58:18.772 - WARNING - Jetty: (2/21) #011at java.util.Arrays.copyOf(Arrays.java:3332) Mar 17 07:58:18.772 - WARNING - Jetty: (3/21) #011at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124) Mar 17 07:58:18.772 - WARNING - Jetty: (4/21) #011at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:649) Mar 17 07:58:18.772 - WARNING - Jetty: (5/21) #011at java.lang.StringBuffer.append(StringBuffer.java:381) Mar 17 07:58:18.772 - WARNING - Jetty: (6/21) #011at org.json.JSONObject.toString(JSONObject.java:1396) Mar 17 07:58:18.773 - WARNING - Jetty: (7/21) #011at com.mendix.m2ee.appcontainer.server.handler.HttpAdminHandler.processAdminException(HttpAdminHandler.java:251) Mar 17 07:58:18.773 - WARNING - Jetty: (8/21) #011at com.mendix.m2ee.appcontainer.server.handler.HttpAdminHandler.handle(HttpAdminHandler.java:152) Mar 17 07:58:18.773 - WARNING - Jetty: (9/21) #011at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) Mar 17 07:58:18.773 - WARNING - Jetty: (10/21) #011at org.eclipse.jetty.server.Server.handle(Server.java:368) Mar 17 07:58:18.773 - WARNING - Jetty: (11/21) #011at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489) Mar 17 07:58:18.773 - WARNING - Jetty: (12/21) #011at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:953) Mar 17 07:58:18.773 - WARNING - Jetty: (13/21) #011at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1014) Mar 17 07:58:18.773 - WARNING - Jetty: (14/21) #011at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:861) Mar 17 07:58:18.773 - WARNING - Jetty: (15/21) #011at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240) Mar 17 07:58:18.774 - WARNING - Jetty: (16/21) #011at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) Mar 17 07:58:18.774 - WARNING - Jetty: (17/21) #011at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:628) Mar 17 07:58:18.774 - WARNING - Jetty: (18/21) #011at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52) Mar 17 07:58:18.774 - WARNING - Jetty: (19/21) #011at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) Mar 17 07:58:18.774 - WARNING - Jetty: (20/21) #011at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) Mar 17 07:58:18.774 - WARNING - Jetty: (21/21) #011at java.lang.Thread.run(Thread.java:745) Mar 17 08:02:52.693 - CRITICAL - M2EE: Java heap space Mar 17 08:02:52.693 - ERROR - M2EE: An error occurred while executing action 'get_all_thread_stack_traces'. Mar 17 08:02:56.985 - ERROR - M2EE: (1/4) com.mendix.m2ee.api.AdminException: JVM Error! Mar 17 08:02:56.985 - ERROR - M2EE: (2/4) #011at com.mendix.m2ee.appcontainer.server.handler.HttpAdminHandler.handle(HttpAdminHandler.java:149) Mar 17 08:02:56.985 - ERROR - M2EE: (3/4) Mar 17 08:02:56.985 - ERROR - M2EE: (4/4) Caused by: java.lang.OutOfMemoryError: Java heap space Mar 17 08:03:06.837 - WARNING - Jetty: / Mar 17 08:03:06.837 - WARNING - Jetty: (1/1) java.lang.OutOfMemoryError: Java heap space Mar 17 08:11:20.857 - CRITICAL - M2EE: An unhandled error occurred in the MxRuntime. Mar 17 08:11:20.857 - CRITICAL - M2EE: (1/1) java.lang.OutOfMemoryError: Java heap space Mar 17 08:11:23.930 - CRITICAL - M2EE: An unhandled error occurred in the MxRuntime. Mar 17 08:11:23.930 - CRITICAL - M2EE: (1/1) java.lang.OutOfMemoryError: Java heap space Mar 17 08:12:38.789 - CRITICAL - ActorSystem: Uncaught fatal error from thread [MxRuntimeSystem-action-dispatcher-114] shutting down ActorSystem [MxRuntimeSystem] Mar 17 08:12:40.279 - CRITICAL - ActorSystem: (1/1) java.lang.OutOfMemoryError: Java heap space Mar 17 08:22:06.543 - CRITICAL - ActorSystem: Uncaught fatal error from thread [MxRuntimeSystem-action-dispatcher-8303] shutting down ActorSystem [MxRuntimeSystem] Mar 17 08:22:06.543 - CRITICAL - ActorSystem: (1/1) java.lang.OutOfMemoryError: Java heap space Mar 17 08:28:53.949 - CRITICAL - ActorSystem: Uncaught fatal error from thread [MxRuntimeSystem-action-dispatcher-80] shutting down ActorSystem [MxRuntimeSystem] Mar 17 08:28:55.425 - CRITICAL - ActorSystem: (1/1) java.lang.OutOfMemoryError: Java heap space   Looking at some metrics:   XAS seems to ''spike'' (not sure how to interpet this) at some points. But it seems to spike during times that the application is not being used (6 in the morning). 

asked 2017-03-17

2 answers