Upload of data snapshot to a Free app / Sandbox

0

Because a Free app used too much data (more than 500 MB, of which half it was ProcessedQueueTask objects that are not being cleanup automatically), the app was bound to be removed entirely. Node cleanup. No way around it. I was forced to get the app on another node.

 

Mx support put me on the track of a fresh upload to a sandbox with a data snapshot. In the end I got it working. And this is what I did.

 

I've tried and tested quite some scenarios, but not all. One of the things that matter, is the amount of data uploaded. In the end, a data snapshot of 4MB did work. Other data snapshots of double that size seemed to work as well, but the app never got out of sleep (by the logs I could see the app was working, I never got a login page though). I would assume this is an issue that can be solved by Mx and larger data sets can be uploaded.

 

I did not upload any files. I do not need that very much and that complicated matters.

 

Here are the steps I took:

• Restore backup from cloud to local db with pgAdmin.
• Use a script to check where your large amounts of data is.

SELECT
    schemaname,
    tablename,
    pg_size_pretty(pg_total_relation_size(schemaname || '.' || tablename)) AS total_size
FROM
    pg_catalog.pg_tables
WHERE
    schemaname NOT IN ('pg_catalog', 'information_schema')
ORDER BY
    pg_total_relation_size(schemaname || '.' || tablename) DESC
LIMIT 10;

This is my result now.

 

When I started, the email table took 85 MB.

 

• I removed as much data as possible. Each time after removing data, I did a Vacuum on the db in pgAdmin: right click the db and choose Maintanace...

 

• Then select FULL and OK.

 

• If you now run the script to see where the data is, it will give you an accurate account.

 

• Next is to get the data in the Built-in db in order to make a data snapshot. At least, that is what I think is the only way to make a data snapshot. In order to do that, set up a configuration in your modeler like this.

 

 

• Remove the deployment dir and the data-snapshot-zip.

 

 

• ... and run the app, with the Built-in db configuration selected. This will load the deployment dir with a HSQLDB with all your data.

• Now, while you're still linked to repository, you're able to Add a snapshot.

 

• Now remove the .git dir, close the app and reopen it again. The app is no longer under version controle. 
• Next is to upload the app to a new node - Upload to Version Control Server.

 

 

• After the upload succeeded, click on Publish. This should do the magic. The code is now in the new repo, completely fresh, and the data is loaded in the new nodes db from the data snapshot.

 

I had to try this many, many times before I succeeded. But still, so much better than typing in everything from scratch.

Posted 2025-04-03

0 comments