java.lang.OutOfMemoryError: GC overhead limit exceeded

0
Hi, I have microflow that needs to iterate over 170000 entities and generate a value for an attribute in each. When I run this microflow over a small subset it's fine, but over the full 170000 entities I eventually get a M2EE error telling me the GC overhead limit has been exceeded. The computer I'm developing on only has 4GB of memory and isn't ideal. Is there a way to force a garbage collection? This microflow will only ever be run once or twice as it's used to initially seed the database. Thanks   An unhandled error occurred in the MxRuntime. -------- java.lang.OutOfMemoryError: GC overhead limit exceeded at java.util.HashMap.newNode(HashMap.java:1742) at java.util.HashMap.putVal(HashMap.java:641) at java.util.HashMap.put(HashMap.java:611) at com.mendix.basis.objectmanagement.MendixObjectImpl.clone(MendixObjectImpl.java:537) at com.mendix.basis.objectmanagement.SchemeManagerImpl.getInstance(SchemeManagerImpl.java:929) at com.mendix.basis.action.user.Retriever$class.com$mendix$basis$action$user$Retriever$$buildObject(Retriever.scala:58) at com.mendix.basis.action.user.Retriever$$anonfun$buildObjects$1.apply(Retriever.scala:53) at com.mendix.basis.action.user.Retriever$$anonfun$buildObjects$1.apply(Retriever.scala:52) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) at scala.collection.Iterator$class.foreach(Iterator.scala:893) at scala.collection.AbstractIterator.foreach(Iterator.scala:1336) at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) at scala.collection.AbstractIterable.foreach(Iterable.scala:54) at scala.collection.TraversableLike$class.map(TraversableLike.scala:234) at scala.collection.AbstractTraversable.map(Traversable.scala:104) at com.mendix.basis.action.user.Retriever$class.buildObjects(Retriever.scala:52) at com.mendix.basis.action.user.Retriever$class.retrieveFromDatabase(Retriever.scala:36) at com.mendix.basis.action.user.RetrieveIdListAction.retrieveFromDatabase(RetrieveIdListAction.scala:14) at com.mendix.basis.action.user.RetrieveIdListAction.com$mendix$basis$action$user$RetrieveIdListAction$$retrieve(RetrieveIdListAction.scala:58) at com.mendix.basis.action.user.RetrieveIdListAction$$anonfun$8.apply(RetrieveIdListAction.scala:50) at com.mendix.basis.action.user.RetrieveIdListAction$$anonfun$8.apply(RetrieveIdListAction.scala:50) at scala.collection.TraversableLike$WithFilter$$anonfun$map$2.apply(TraversableLike.scala:683) at scala.collection.immutable.Map$Map1.foreach(Map.scala:116) at scala.collection.TraversableLike$WithFilter.map(TraversableLike.scala:682) at com.mendix.basis.action.user.RetrieveIdListAction.executeAction(RetrieveIdListAction.scala:50) at com.mendix.basis.action.user.RetrieveIdListAction.executeAction(RetrieveIdListAction.scala:14) at com.mendix.systemwideinterfaces.core.UserAction.execute(UserAction.java:50) at com.mendix.basis.actionmanagement.CoreActionHandlerImpl.doCall(CoreActionHandlerImpl.scala:73) at com.mendix.basis.actionmanagement.CoreActionHandlerImpl.call(CoreActionHandlerImpl.scala:53) at com.mendix.core.actionmanagement.CoreAction.call(CoreAction.java:51) at com.mendix.basis.actionmanagement.ActionManager$1.execute(ActionManager.java:170)  
asked
6 answers
5

In case of such amounts (>10k) you need microflow that processes the data in chunks. Have a look at this blog

answered
4

You must use batches to do this. See the documentation here: https://docs.mendix.com/howto40/retrieve-and-manipulate-batches-of-objects

Strange that the Mx7 update of this how to is not findable. But it still is true.

Regards,

Ronald

 

answered
1

Hi Robert,

As the other answers already state, batching is the way to go here. I would even recommend considering the ProcessQueue for such large computations, then you can also run the batches asynchronously https://appstore.home.mendix.com/link/app/393/Mendix/Process-Queue

-Andrej

answered
1

Of course, batches are needed as is mentioned above. But, with this amount of records, if it is really needed, only batching will not solve your issue. If everything is done within that Microflow, Mendix will hold all 170.000 records in that microflow until the Microflow is done. This is because Mendix should be able to do a full rollback, once something goes wrong. A way to free up memory, is to run each iteration in a separate Java transaction. Add after each iteration the CommunityCommons.EndTransaction and CommunitiyCommons.StartTransaction Java actions to make sure memory is released. 

Of course only do this if it is really needed. A full rollback is in this scenario not possible anymore.

Other option, but way more time consuming, is the addition of the ProcessQueue module. Add for every iteration the QueuedAction and trigger this QueuedAction from the ProcessQueue engine. This will also fire each QueuedAction in a separate Java transaction, hence not clogging the memory.

 

answered
0

To learn more about data and especially batches, try the Working with Data course:

Section 3 focuses specifically on batches.

answered
0

Robert,

The quick and dirty solution (since this microflow  will only be used once or twice).

Add a boolean attribute to your entity - call it Updated or Processed or something similar.

In your microflow:

  • Retrieve 10,000 records to process where your new attribute = false().  ( get 10,000 records using Range Custom in the retrieve action)
  • While processing those records in the loop, set your new attribute to true

Now you can run this microflow as many times as needed to update all of your records.  You can run it from a web page or via a scheduled event.

Once all the records are updated, you can remove the boolean attribute from your entity.

Note:  best practice is to commit the whole list of objects after the loop is complete instead of committing each object in the loop.  You may already be doing this.

Hope that helps,

Mike

answered