Question

Timeout and Feedback size Issue - MX 7.5.1

0

I am running an import of data process that takes quite a long time to process. I am using the blocking Progress Bar within the action button settings and it is set as synchronous. The process creates numerous non-persistent objects based on a Json transformation and then processes these through into the apps persistent entities. The process takes quite a bit longer than I would expect - roughly 23 minutes to process 2500 lines of data (0.5Mb) and is causing a few issues. 1. The following error is thrown client side during the process which causes a pop up error being displayed [error] 166#0: *2873 upstream timed out (110: Connection timed out) while reading response header from upstream, 2. The following is also thrown in the log WARNING - WebUI: Feedback size of 14846 exceeds threshold of 5000 items which can result in severe performance problems. I am currently cutting the size of the import files from 10,000 down to 2500 lines so that the process finishes, these could potentially be far bigger. Resolving the above errors would be a help and any advice on whether the timing to process is reasonable or not would be helpful too. To help give an idea of the process, I am using a custom retrieve, currently set at 100 records in an outer loop, an inner loop processes the 100 records and on each inner iteration it does a Json transformation of the passed file creating all the non-persistent entities, processes the information based on business logic and creates the persistent objects required by the app with a commit on these and finally does a delete on the non-persistent top entity to help free up memory.

asked 2017-10-11

Tim Threadingham

3 answers

Ronald Catersels · Answer 1 · 2017-10-11

I would start with setting the microflow to asynchronous as you should always do with long running microflows. Furthermore I would do batch processing of the objects if this is possible. See the documentation here: https://docs.mendix.com/howto40/retrieve-and-manipulate-batches-of-objects.

Regards,

Ronald

[EDIT]

You could try the community commons Start transaction and End transaction for the inner loop. The timeout I am not sure off why this is happening. It might be solved by using the call asynchronous. Otherwise file a bugreport for that part.

Tim Threadingham · Answer 2 · 2017-10-13

Similar to the Start and end transaction - is there a way of creating a new Context and passing this back to my microflow. I am thinking along the lines of then being able to do a start and end transaction on my new context for each of the batches of my inner loop

eg retrieve batch of records --> call Java action to create new Context --> call java start transaction --> process records batch -- > call java action end transaction

Is this feasible, does it make sense?

Tim Threadingham · Answer 3 · 2017-10-16

Thanks for the replies and pointers guys.

My final resolution was to follow the pointer from Erwin - so for anyone else interested,

I split out my processing on my inner loop - the loop doing the batch process of 100 records per time - and separated out the actual process into another microflow. I then reworked my loop a bit and then created a new Java process to take the arguments of my new processing microflow, and the object I am using to create all my new entities. Within my new java process I have created a new context and called a start transaction on this. I have then used the Core.execute() to run my new process microflow passing it my object argument sent via the java action. After this has run I then call the end transaction on my new context and return out of the Java action. So in effect my original inner process loop now has a minimal amount of code in it and calls my new Java process for each record within the loop.

This has completely resolved the amount of processing time that it was originally taking down to about two minutes