Application memory warning

0
Hi experts, I got following in my log can any one help what all may possible reasons for the issue. 2023-02-07T02:16:22.763433 [APP/PROC/WEB/0] WARNING - ConnectionBus: Database connections: 1 active, 24 idle. 2023-02-07T02:16:22.763436 [APP/PROC/WEB/0] WARNING - ConnectionBus: Database connections: 1 active, 24 idle. 2023-02-07T02:16:22.763500 [APP/PROC/WEB/0] WARNING - ConnectionBus: Database connections: 1 active, 24 idle. 2023-02-07T02:16:22.763555 [APP/PROC/WEB/0] WARNING - ConnectionBus: Database connections: 1 active, 24 idle. 2023-02-07T02:16:22.763572 [APP/PROC/WEB/0] WARNING - ConnectionBus: Database connections: 1 active, 24 idle. 2023-02-07T02:16:22.763581 [APP/PROC/WEB/0] WARNING - ConnectionBus: Database connections: 1 active, 24 idle. 2023-02-07T02:16:22.763641 [APP/PROC/WEB/0] WARNING - ConnectionBus: Database connections: 1 active, 24 idle. 2023-02-07T02:16:22.763659 [APP/PROC/WEB/0] WARNING - ConnectionBus: Database connections: 1 active, 24 idle. 2023-02-07T02:16:22.763722 [APP/PROC/WEB/0] ERROR - ConnectionBus: Opening JDBC connection to Some(hdfhsjk4iomoio208v.cvbgouwovei.ap-northeast-1.rds.amazonaws.com:5432) failed with SQLState: 08001 Error code: 0 Message: Connection attempt timed out. Retrying...(1/4) 2023-02-07T02:16:22.763729 [APP/PROC/WEB/0] ERROR - ConnectionBus: Opening JDBC connection to Some(hdfhsjk4iomoio208v.cvbgouwovei.ap-northeast-1.rds.amazonaws.com:5432) failed with SQLState: 08001 Error code: 0 Message: Connection attempt timed out. Retrying...(1/4) 2023-02-07T02:16:22.763767 [APP/PROC/WEB/0] ERROR - ConnectionBus: Opening JDBC connection to Some(hdfhsjk4iomoio208v.cvbgouwovei.ap-northeast-1.rds.amazonaws.com:5432) failed with SQLState: 08001 Error code: 0 Message: Connection attempt timed out. Retrying...(1/4) 2023-02-07T02:16:22.763779 [APP/PROC/WEB/0] ERROR - ConnectionBus: Opening JDBC connection to Some(hdfhsjk4iomoio208v.cvbgouwovei.ap-northeast-1.rds.amazonaws.com:5432) failed with SQLState: 08001 Error code: 0 Message: Connection attempt timed out. Retrying...(1/4) 2023-02-07T02:16:22.763887 [APP/PROC/WEB/0] ERROR - ConnectionBus: Opening JDBC connection to Some(hdfhsjk4iomoio208v.cvbgouwovei.ap-northeast-1.rds.amazonaws.com:5432) failed with SQLState: 08001 Error code: 0 Message: Connection attempt timed out. Retrying...(1/4) 2023-02-07T02:16:22.763902 [APP/PROC/WEB/0] ERROR - ConnectionBus: Opening JDBC connection to Some(hdfhsjk4iomoio208v.cvbgouwovei.ap-northeast-1.rds.amazonaws.com:5432) failed with SQLState: 08001 Error code: 0 Message: Connection attempt timed out. Retrying...(1/4) 2023-02-07T02:16:22.763916 [APP/PROC/WEB/0] ERROR - ConnectionBus: Opening JDBC connection to Some(hdfhsjk4iomoio208v.cvbgouwovei.ap-northeast-1.rds.amazonaws.com:5432) failed with SQLState: 08001 Error code: 0 Message: Connection attempt timed out. Retrying...(1/4) 2023-02-07T02:16:22.763935 [APP/PROC/WEB/0] WARNING - ConnectionBus: Database connections: 1 active, 24 idle. 2023-02-07T02:16:22.763948 [APP/PROC/WEB/0] ERROR - ConnectionBus: Opening JDBC connection to Some(hdfhsjk4iomoio208v.cvbgouwovei.ap-northeast-1.rds.amazonaws.com:5432) failed with SQLState: 08001 Error code: 0 Message: Connection attempt timed out. Retrying...(1/4) 2023-02-07T02:16:22.763989 [APP/PROC/WEB/0] ERROR - ConnectionBus: Opening JDBC connection to Some(hdfhsjk4iomoio208v.cvbgouwovei.ap-northeast-1.rds.amazonaws.com:5432) failed with SQLState: 08001 Error code: 0 Message: Connection attempt timed out. Retrying...(1/4) 2023-02-07T02:16:22.764027 [APP/PROC/WEB/0] ERROR - ConnectionBus: Opening JDBC connection to Some(hdfhsjk4iomoio208v.cvbgouwovei.ap-northeast-1.rds.amazonaws.com:5432) failed with SQLState: 08001 Error code: 0 Message: Connection attempt timed out. Retrying...(1/4) 2023-02-07T02:16:22.764046 [APP/PROC/WEB/0] ERROR - ConnectionBus: Opening JDBC connection to Some(hdfhsjk4iomoio208v.cvbgouwovei.ap-northeast-1.rds.amazonaws.com:5432) failed with SQLState: 08001 Error code: 0 Message: Connection attempt timed out. Retrying...(1/4) 2023-02-07T02:16:22.764105 [APP/PROC/WEB/0] ERROR - ConnectionBus: Opening JDBC connection to Some(hdfhsjk4iomoio208v.cvbgouwovei.ap-northeast-1.rds.amazonaws.com:5432) failed with SQLState: 08001 Error code: 0 Message: Connection attempt timed out. Retrying...(1/4) 2023-02-07T02:16:22.764115 [APP/PROC/WEB/0] WARNING - ConnectionBus: Database connections: 1 active, 24 idle. 2023-02-07T02:16:22.764158 [APP/PROC/WEB/0] ERROR - ConnectionBus: Opening JDBC connection to Some(hdfhsjk4iomoio208v.cvbgouwovei.ap-northeast-1.rds.amazonaws.com:5432) failed with SQLState: 08001 Error code: 0 Message: Connection attempt timed out. Retrying...(1/4) 2023-02-07T03:36:45.587555 [APP/PROC/WEB/0] 2023/02/07 03:36:45 [error] 278#0: *90262 upstream timed out (110: Connection timed out) while connecting to upstream, client: 144.15.255.227, server: _, request: "POST /rest/v1/apicall/search HTTP/1.1", upstream: "http://127.0.1.1:8081/rest/apicall/search", host: "irack.mendixcloud.com" 2023-02-07T03:36:56.847243 [APP/PROC/WEB/0] 2023/02/07 03:36:56 [error] 278#0: *90265 upstream timed out (110: Connection timed out) while connecting to upstream, client: 165.225.234.189, server: _, request: "POST /xas/ HTTP/1.1", upstream: "http://127.0.0.1:8081/xas/", host: "irack.mendixcloud.com", referrer: "http://irack.mendixcloud.com/index.html" 2023-02-07T03:37:20.821533 [APP/PROC/WEB/0] WARNING - ConnectionBus_Queries: Query executed in 73 seconds and 658 milliseconds: SELECT "sharedmasterdata$bu"."id", 2023-02-07T03:37:20.821585 [APP/PROC/WEB/0] "sharedmasterdata$bu"."endescription" 2023-02-07T03:37:20.821657 [APP/PROC/WEB/0] FROM "sharedmasterdata$bu" 2023-02-07T03:37:20.821670 [APP/PROC/WEB/0] WHERE "sharedmasterdata$bu"."id" IN (SELECT "medicalevent$medicalevent_bu"."sharedmasterdata$buid" 2023-02-07T03:37:20.821673 [APP/PROC/WEB/0] FROM "medicalevent$medicalevent_bu" 2023-02-07T03:37:20.821746 [APP/PROC/WEB/0] WHERE "medicalevent$medicalevent_bu"."medicalevent$medicaleventid" IN (661466195274947512, 66146243527494753, 6435146619527494596, 65454619527494592, 6544146619527494584, 66146619527494545, 66146619527494544, 66146619527494539, 66146619527494533, 66146619527494320)) 2023-02-07T03:37:28.175067 [APP/PROC/WEB/0] 2023/02/07 03:37:28 [error] 278#0: *90270 upstream timed out (110: Connection timed out) while connecting to upstream, client: 165.225.134.189, server: _, request: "POST /xas/ HTTP/1.1", upstream: "http://127.0.0.1:8081/xas/", host: "irack.mendixcloud.com", referrer: "http://irack.mendixcloud.com/index.html" 2023-02-07T03:37:28.175180 [APP/PROC/WEB/0] 2023/02/07 03:37:28 [error] 278#0: *90271 upstream timed out (110: Connection timed out) while connecting to upstream, client: 165.225.243.189, server: _, request: "POST /xas/ HTTP/1.1", upstream: "http://127.0.0.1:8081/xas/", host: "irack.mendixcloud.com", referrer: "http://irack.mendixcloud.com/index.html" 2023-02-07T03:38:52.716767 [APP/PROC/WEB/0] 2023/02/07 03:38:52 [error] 278#0: *90280 upstream timed out (110: Connection timed out) while connecting to upstream, client: 165.225.156.98, server: _, request: "POST /xas/ HTTP/1.1", upstream: "http://127.0.0.1:8081/xas/", host: "irack.mendixcloud.com", referrer: "http://irack.mendixcloud.com/index.html" 2023-02-07T03:38:58.143098 [APP/PROC/WEB/0] 2023/02/07 03:38:58 [error] 278#0: *90282 upstream timed out (110: Connection timed out) while connecting to upstream, client: 144.15.255.227, server: _, request: "POST /rest/v1/apicall/search/search HTTP/1.1", upstream: "http://127.0.1.1:8081/rest/apicall/search", host: "irack.mendixcloud.com" 2023-02-07T03:39:47.297872 [APP/PROC/WEB/0] WARNING: Unable to get critical logs count from runtime 2023-02-07T03:39:52.721792 [APP/PROC/WEB/0] 2023/02/07 03:39:52 [error] 278#0: *90280 upstream timed out (110: Connection timed out) while connecting to upstream, client: 165.225.165.78, server: _, request: "POST /xas/ HTTP/1.1", upstream: "http://127.0.0.1:8081/xas/", host: "irack.mendixcloud.com", referrer: "http://irack.mendixcloud.com/index.html" 2023-02-07T03:39:52.730075 [APP/PROC/WEB/0] 2023/02/07 03:39:52 [error] 278#0: *90291 upstream timed out (110: Connection timed out) while connecting to upstream, client: 165.225.236.98, server: _, request: "POST /xas/ HTTP/1.1", upstream: "http://127.0.0.1:8081/xas/", host: "irack.mendixcloud.com", referrer: "http://irack.mendixcloud.com/index.html" 2023-02-07T03:39:52.734572 [APP/PROC/WEB/0] 2023/02/07 03:39:52 [error] 278#0: *90292 upstream timed out (110: Connection timed out) while connecting to upstream, client: 165.225.116.95, server: _, request: "POST /xas/ HTTP/1.1", upstream: "http://127.0.0.1:8081/xas/", host: "irack.mendixcloud.com", referrer: "http://irack.mendixcloud.com/index.html" 2023-02-07T03:39:52.738897 [APP/PROC/WEB/0] 2023/02/07 03:39:52 [error] 278#0: *90290 upstream timed out (110: Connection timed out) while connecting to upstream, client: 165.225.116.98, server: _, request: "POST /xas/ HTTP/1.1", upstream: "http://127.0.0.1:8081/xas/", host: "irack.mendixcloud.com", referrer: "http://irack.mendixcloud.com/index.html" 2023-02-07T03:40:26.993586 [APP/PROC/WEB/0] # 2023-02-07T03:40:26.993627 [APP/PROC/WEB/0] # java.lang.OutOfMemoryError: Java heap space 2023-02-07T03:40:26.993643 [APP/PROC/WEB/0] # -XX:OnOutOfMemoryError="kill -s USR2 7" 2023-02-07T03:40:26.993646 [APP/PROC/WEB/0] # Executing /bin/sh -c "kill -s USR2 7"... 2023-02-07T03:40:26.995900 [APP/PROC/WEB/0] INFO: Waiting for the application to shutdown... 2023-02-07T03:40:42.018040 [APP/PROC/WEB/0] WARNING: The application did not shutdown by itself... 2023-02-07T03:40:42.018131 [APP/PROC/WEB/0] WARNING: Waiting for runtime process failed: [Errno 10] No child processes 2023-02-07T03:40:42.018297 [APP/PROC/WEB/0] INFO: Waiting for the JVM process to disappear... 2023-02-07T03:40:52.031059 [APP/PROC/WEB/0] WARNING: The application process seems not to respond to any command or signal. 2023-02-07T03:40:52.031170 [APP/PROC/WEB/0] WARNING: Could not terminate runtime 2023-02-07T03:41:16.623895 [HEALTH/0] Liveness check unsuccessful: failed to make TCP connection to 10.255.209.43:8080: dial tcp 10.255.209.43:8080: connect: connection refused 2023-02-07T03:41:16.623902 [CELL/0] Container became unhealthy 2023-02-07T03:41:16.634539 [CELL/SSHD/0] Exit status 0  
asked
1 answers
1

ERROR - ConnectionBus: Opening JDBC connection to Some(hdfhsjk4iomoio208v.cvbgouwovei.ap-northeast-1.rds.amazonaws.com:5432) failed with SQLState: 08001 Error code: 0 Message: Connection attempt timed out.

 

indicates that the Server where your SQL DB is located cant be reached,

 

2023-02-07T03:40:26.993627 [APP/PROC/WEB/0] # java.lang.OutOfMemoryError: Java heap space 

is the more critical error, this indicates that your applilcation ran out of memory. Maybe check the Metrics page of your Application. What container Size are you using? Seems like you had load spikes or you need to improve the apps performance. This Error caused your application to crash and shut down. 

 

 

answered
0

do you have a lot of objects in memory, like a retrieve all?

or do you have a lot of concurrent actions running?

maybe you have to look towards that.

 

It is strange that there are idle connections while connecting is not possible. 

thats becasue that the app is getting out of memory, due to the “java heap space” error message.

answered