export large dataset to csv

2
In nightly events we write database log lines to a FileDocument, in CSV format. Currently this is done by building one large String and then using CommunityCommons.StringToFile. I am concerned that this may cause memory issues, as typically over 100.000 lines are written to the file. Is this concern justly, or should I not bother about memory issues? I was already looking for a solution but it does not seem possible to stream the text to the FileDocument. In case there are other solutions, please let me know, I am very interested.
asked
1 answers
2

In a Java action use a temporary file and save that in the filedocument contents. Beware of the type of the used streams otherwise it will still use memory. If the storage of the filedocuments is in another storageprovider/cloud like amazon it will probably always use memory.

Another option is to limit the filesize and create more filedocuments.

answered