0

Hi everyone,   I am encountering an issue while processing a "large" list of non-persistable entities to remove duplicates.   Following situation: The entity includes about 10 attributes, the combination of three of those attributes has to be unique. In hope of improving the process and reading some recommendations on the forum I have created an additional "Identifier" attribute which concatenates the three attributes into one attribute. (Before I was using the find by expression looking for each attribute).   E.g., Entity: Attr1, Attr2, Attr3, Attr4, Attr5, ... -> IdentifierAttr = Attr1 + Attr2 + Attr3   The records are getting created in a submicroflow that I am calling before my "post processing" to remove the duplicates (the Identifier is set during creation).    In my post processing I am iterating over this list of NPE´s (around 150000 records) and identifying the duplicates in the "standard" way (see screenshot). The result should be something around 110000 records. I am not sure why but this post processing sub microflow is taking forever (> 30-60 min). The creation of these 150k records is much more complex and takes only about 5-6 min.    Additionally I am getting a warning from the connection bus that "Transaction XYZ aborted due to timeout" even though the microflow is called asynchronously and also the microflow is still running after that warning.   Am I missing something? Is there a recommended approach to improve this process or an explanation for the extended duration?   Your insights would be greatly appreciated, thanks in advance!   Best regards, Jan  

asked 2024-09-03

1 answers