CSV Services - Importing to cloud node gives a timeout after a minute

0
Hi, I'm using CSV Services (https://marketplace.mendix.com/link/component/1911) to copy the content of a SQL Server db to a Mx db. I've downloaded CSV files from the SQL Server db and upload them in the Mx app with CSV Services. Locally, this works fine. I only need to keep the maximum runtime into account and stay within the limit of about 2 hours. As some files are many millions of records, this is a hassle. Instead of locally importing the CSV files, I now tried it on a cloud node. The problem I'm facing is that a timeout message is given after a minute or so and only part of the CSV file is processed. Below you find the command and the output with the timeout at the bottom. Anyone an idea why this is happening and how to solve it? Thanks Toon curl -v -X POST -H "Content-Type: text/csv" https://MxAdmin:xxxxxxxxx@xxxxxxxxxx-accp.mendixcloud.com/csv/PartsStub/CardLine --data-binary "@CardLine 11.csv" --no-keepalive Note: Unnecessary use of -X or --request, POST is already inferred. * Trying 3.127.11.115... * TCP_NODELAY set * Connected to xxxxxxxxxxxxx-accp.mendixcloud.com (3.127.11.115) port 443 (#0) * schannel: SSL/TLS connection with xxxxxxxxxxxxxxxx-accp.mendixcloud.com port 443 (step 1/3) * schannel: checking server certificate revocation * schannel: sending initial handshake data: sending 199 bytes... * schannel: sent initial handshake data: sent 199 bytes * schannel: SSL/TLS connection with xxxxxxxxxxxxxx-accp.mendixcloud.com port 443 (step 2/3) * schannel: failed to receive handshake, need more data * schannel: SSL/TLS connection with xxxxxxxxxxxx-accp.mendixcloud.com port 443 (step 2/3) * schannel: encrypted data got 3216 * schannel: encrypted data buffer: offset 3216 length 4096 * schannel: sending next handshake data: sending 93 bytes... * schannel: SSL/TLS connection with xxxxxxxxxxx-accp.mendixcloud.com port 443 (step 2/3) * schannel: encrypted data got 290 * schannel: encrypted data buffer: offset 290 length 4096 * schannel: SSL/TLS handshake complete * schannel: SSL/TLS connection with xxxxxxxxxxxxxxxx-accp.mendixcloud.com port 443 (step 3/3) * schannel: stored credential handle in session cache * Server auth using Basic with user 'MxAdmin' > POST /csv/PartsStub/CardLine HTTP/1.1 > Host: xxxxxxxxxx-accp.mendixcloud.com > Authorization: Basic xxxxxxxxxxxxxxxxxxxx > User-Agent: curl/7.55.1 > Accept: */* > Content-Type: text/csv > Content-Length: 35941043 > Expect: 100-continue > * schannel: client wants to read 102400 bytes * schannel: encdata_buffer resized 103424 * schannel: encrypted data buffer: offset 0 length 103424 * schannel: encrypted data got 54 * schannel: encrypted data buffer: offset 54 length 103424 * schannel: decrypted data length: 25 * schannel: decrypted data added: 25 * schannel: decrypted data cached: offset 25 length 102400 * schannel: encrypted data buffer: offset 0 length 103424 * schannel: decrypted data buffer: offset 25 length 102400 * schannel: schannel_recv cleanup * schannel: decrypted data returned 25 * schannel: decrypted data buffer: offset 0 length 102400 < HTTP/1.1 100 Continue * We are completely uploaded and fine * schannel: client wants to read 102400 bytes * schannel: encrypted data buffer: offset 0 length 103424 * schannel: encrypted data got 474 * schannel: encrypted data buffer: offset 474 length 103424 * schannel: decrypted data length: 445 * schannel: decrypted data added: 445 * schannel: decrypted data cached: offset 445 length 102400 * schannel: encrypted data buffer: offset 0 length 103424 * schannel: decrypted data buffer: offset 445 length 102400 * schannel: schannel_recv cleanup * schannel: decrypted data returned 445 * schannel: decrypted data buffer: offset 0 length 102400 < HTTP/1.1 504 Gateway Timeout < Server: nginx < Date: Fri, 26 Mar 2021 15:52:20 GMT < Content-Type: text/html < Content-Length: 160 < Connection: keep-alive < X-Frame-Options: sameorigin < X-Vcap-Request-Id: bfc5643a-c880-4656-77a0-05a213544673 < Strict-Transport-Security: max-age=31536000 < <html> <head><title>504 Gateway Time-out</title></head> <body> <center><h1>504 Gateway Time-out</h1></center> <hr><center>nginx</center> </body> </html> * Connection #0 to host xxxxxxx-accp.mendixcloud.com left intact  
asked
2 answers
0

Were you ever able to resolve your issue? I’ve been dealing with the same thing for a month. I’m getting a 504 Server timeout that is not coming from Cloudflare and likely not coming from azure. It’s happening on any process that takes longer than 60 seconds. Even on uploading a 60mb file to compile into sql and send to Azure in batches of 1k records. We need to be able to handle up to 1million records on a csv

answered
0

Did you change it to running async? Typically the webserver will time out for a microflow running synchronously if it runs for more than 60-90 seconds. 

answered