Speed up REST data import

0
Hi all   I am retrieving a lot of data from a REST-api. I figured I definitely need to make the import process faster but I’m not sure how.   I have to make three calls to the api. The first retrieves about 300 objects, the second about 7’000 and the third 100’000. I then import those with an import mapping where the latter two need to create a reference to the first object if it’s ID is in the response (First call are projects and second and third are not billed activities and billed activities on the project).   Everything works how I want to but the problem is it takes almost 2-3 Minutes to retrieve and create those objects if not found with a key. Does anybody know how I could improve the speed in this? (If it’s even possible with this amount of data)   Thanks
asked
1 answers
0

Do you need this data immediately when the user requests it? And how important is it that the data is entirely up to date?

If at all possible, I would consider an alternative approach:

Request the data periodically depending on how up to date it has to be (e.g. daily, hourly, every 15 minutes) and save it in the database. This way the user won't have to wait for the processing time. Any processing from the calls could be done in the background, e.g. via a process queue.

answered