I would start with setting the microflow to asynchronous as you should always do with long running microflows. Furthermore I would do batch processing of the objects if this is possible. See the documentation here: https://docs.mendix.com/howto40/retrieve-and-manipulate-batches-of-objects.
You could try the community commons Start transaction and End transaction for the inner loop. The timeout I am not sure off why this is happening. It might be solved by using the call asynchronous. Otherwise file a bugreport for that part.
Similar to the Start and end transaction - is there a way of creating a new Context and passing this back to my microflow. I am thinking along the lines of then being able to do a start and end transaction on my new context for each of the batches of my inner loop
eg retrieve batch of records --> call Java action to create new Context --> call java start transaction --> process records batch -- > call java action end transaction
Is this feasible, does it make sense?
Thanks for the replies and pointers guys.
My final resolution was to follow the pointer from Erwin - so for anyone else interested,
I split out my processing on my inner loop - the loop doing the batch process of 100 records per time - and separated out the actual process into another microflow. I then reworked my loop a bit and then created a new Java process to take the arguments of my new processing microflow, and the object I am using to create all my new entities. Within my new java process I have created a new context and called a start transaction on this. I have then used the Core.execute() to run my new process microflow passing it my object argument sent via the java action. After this has run I then call the end transaction on my new context and return out of the Java action. So in effect my original inner process loop now has a minimal amount of code in it and calls my new Java process for each record within the loop.
This has completely resolved the amount of processing time that it was originally taking down to about two minutes