Call Rest (GET) Action is Extremely Slow

0
Hi,   Call Rest (GET) action is extremely slow. When I debug, I can see that it takes minutes. On the other hand, I can pull the data in PostMan without any problems in 5 seconds. I've shared the settings for my call rest action below. Also, when mapping, I use find by key, and the "if no object was found" option is set to "create."  
asked
2 answers
2

As you mention 'Find by key', you have to be aware that this will result into 55.000 separate SELECT queries to the database and that a quite large transaction is started when creating/updating that amount of persistent entities.

I always recommend to split integrations in 3 actions:

  1. REST Call (to HTTPResponse)
  2. Import Mapping (to NPE) and
  3. Processing (to PE in subflow)

In this case, the biggest gain you can make is probably to cache the list of students in DB using a single retrieve and then use a find in the loop while processing the NPE objects.

Good luck!

answered
0

It could be that the import mapping is taking a long time to run. To verify this, you could store the output of the REST action in a string and apply the import mapping in a seperate action after the REST call.

 

If it's indeed the import mapping, my first suggestion would be to check what fields you are using as keys in the import mapping and make to create indexes on those attribute. I've also faced slow import mappings due to tables with  > 1 million records not properly being indexed.

 

How big is the dataset you are retrieving and how many Students are in the database?

answered