Question

Best performance option for archiving data?

0

We have an application with hundred thousands of objects, with loads of assocations. We are running into performance issues. 80-90% of this data is not relevant anymore, so we want to implement archiving. In the Community Best Practices for App Performance it states the following: "Consider archiving data if your volume grows too large and you do not need all the data all the time. You can even consider creating two identical entities, one with the data currently being used, and the other with all the data that is only used for reporting or other historic reasons." Would this be the best performance choice for archiving? The easiest solution would be to just apply an Archived boolean, but I'm not sure how much this would increase the performance. The only con with archiving data into a seperate entity lies with the associations. Since we have loads of associated objects, we would then need to also make seperate archivied entities for these objects.

asked 2025-04-22

Kurt Rosa Bian

4 answers

Rachana BP · Answer 1 · 2025-04-22

Hi,

We had done archiving once and we didnt create any separate entity for the same. using the flag -> we transported the data to S3 and stored it over their and cleared the flagged data in mendix. Please check if this method works for you.

Jelle van Leeuwen · Answer 2 · 2025-04-22

Hello Kurt,

There are a lot of possibilities to archive the data:

If you would like to fully archive it in mendix you can try the approach as mendix describe, however in that case you data stays in the same database, so if you have database size issues this will not resolve it.

Other option is for example the option Rachana proposes but with this option you need to buy an extra S3 database/bucket to store your archive, benefit is that it will decrease your database size in mendix drastically. To access the archive you need to make a connection to the S3 bucket there are modules for that.

Other option that i have seen are simple csv exports to a location outside mendix. Problem with this is that you cannot access the data that simply again within mendix.

Hope this gives some insight,

Good Luck!

Dave Lina · Answer 3 · 2025-04-22

Sure, the seperate entity can help you a lot but can be difficult or time consuming to apply.

We also created the Archived boolean in combination with an index and this provides some performance gain. This is our approach for the time being, but I can't state that we have the same case. We are talking about tables having around 250.000 records, but also a lot of associations.

Also, you can set your grids (if you still have datagrid 1 etc) to not calculate the amount of records what can be a huge performance gain.

Alternative option is to retrieve your records by OQL. This can be tricky but it can help you a lot with performance.

There are multiple modules that can you help you with this.

Thijs Oosting · Answer 4 · 2025-04-22

Depends on your definition of archiving, your data structure and requirements in usage.

There are already some good options shared, another one could be a different entity called Archive where you store your data in a string value by using JSON and an association towards the main entity. Than when viewing an archived version you can use a NPE to display the required data.