Hello all, In my application i use the CSV import widget . My purpose here is , I want all the lines read and imported by system in my CSV file. I made a non- persistable entity in domain model first in order to read all the lines. If the lines doesnt exist in the system, i create/update an object in my system. I have almost 1 million lines of data(Text fields) that needs to be checked and downloaded everyday. So it is fact that this large amount of can not be imported immedietly. Thats why i have been using task queue and trying to import my data 50000 by 50000 lines. In the Task Queue there is an option as ‘Thread’. When increase the amount of Thread, i recieved an error below ‘CRITICAL: Your application's virtual machine ran out of memory and died’ If i made the thread as lower numbers, operation is finished and recieved an eror called ‘Connection Reset’. And those transactions are keep resuming and never finishes. Can someone help fort his case what could be the best option for the above case?
asked
Mert Tirmandioglu
1 answers
0
If you need to read and check 1 million lines every day it would be wise to build something in java that performs a check without the need to import the data and then do the check. In the past I have build a small java action that compared the current csv file with the previous csv file. Then create 2 new csv file, one that contained changed and new data and one that stored the deleted data. Then you import these files that will probably be much smaller assuming that the majority of the lines will not change. This prevents the issues with the timeouts. To check the lines I've used a checksum to compare the previous and current files this worked for files with millions of records as a charm.