Large Data Directory

2
What is considered a large data directory? *EDIT : * "what things should I keep in mind when my application grows?" – Achiel :) Our Application\data directory has grown to 80GB of which the Application\data\files directory takes 78.2GB. It contains 2 000 000 files. Our system has been in full production since February 2012, should I be concerned? We have been experiencing some performance issues, which I am currently investigating and I am wondering whether this could be the problem.
asked
1 answers
3

I do think that 2 million files is a large directory structure, and retrieving files for images or attachments may be affected by file access times. These files are not stored in the database, so you can't tune that, so your only options are to try to manage your file numbers...

One obvious application design feature is to make sure that you have set up the correct delete behaviour in your domain model, so if a record gets deleted, any linked images or files also get deleted.

If the issue is the disk space used, rather than the number of files, look at restricting the size you allow to be uploaded for images and attachments.

The other area to look at with a high-volume application is archiving strategy for your data. Do you need to retain all historical data in the application, or can you archive old records to another storage medium, or even to a separate Mendix server instance.

I routinely design applications with 'internal' archiving, where the number of records in the main active tables is kept to a manageable number by moving old records to separate tables (entities). Unfortunately in Mendix this does not help issues with the storage of file and image attachments.

You can specify where to store these in the runtime configuration (so you could store them on a high performance SAN) but currently you can only specify a single file path. Perhaps there should be an enhancement request to be able to specify more that one location to save files/images, so old data files can be stored in a separate location to current ones.

answered