Scheduled events with Task queues

0
Hi All, Following is the my use case. step-1 Need to bring data from external database to intermediate entities. step-2 Need to update the status on external data base for successful migrated records. step3- Need to process the data from the intermediate entities to actual entities after updating the status on the external database.(All intermediate entities data depends on each other to process the data ) How to efficiently create scheduled events for above 3 steps without overlapping with each other and with out loosing any data using task queues as data set is very large ?   TIA    
asked
1 answers
0

I would create a scheduled event for step 1 that ends in an action that adds the records in a taskqueue. Adding the records could be done with an extra entity that has the imported records associated to it. The  process the records with the microflow in the taskqueue with a batch pattern and end this with a new record in the taskqueue that processes all the records associated to the extra entity. 

THis ensures the process is sequential and can only overlap with e new batch perfroming the three steps if the processing of the previous batch has not been finished. To prevent this you could create a control record in an entity that is set at the start with the scheduled event and updated when steps 3 has finished, then yoou can add a check in the scheduled event to see if this can be executed or needs to wait for the processing to finish. Using the endtransaction java action from community commons might be helpful depending on the scheduled events interval.

answered