What I usually do with these import functionalities, is doing it in 2 steps. First import all the data in an Importdata entity. Once all data is imported, start a microflow to process all these imported records, and set a processed-flag. This processing should be done in batches and with an end-transaction after each batch.
When does your processing stop? During the import of the file, or the processing of the imported entries?
Hallo Rogier,
The file is saved in a import data entity and processed in batches. Process saves a part of the file in a import data entity → process the import data entity in batches → once finished delete the import data entity and start importing from the last line read – process this until csv = empty.
The headschatcher from me is that the entire process seems to work fine when I upload the file manually (using the file manager widget and triggering the same MF). Lines that are not being processed partially when file is pulled from sftp are being processed completely when csv is uploaded manually.