Large mpr file - GIT LFS support

3
Hello, We're currently experiencing a critical issue that is preventing us from migrating from Mendix SVN to our private Github enterprise repository. Explanation of the issue: On one hand, we have the mendix framework that tracks any changes made in a project into a single .mpr file located at the root of the project. This file becomes bigger and bigger after any commit, file changes, file additions etc... On one of our project, this .MPR file is currently weighting 110MB. Its size cannot be reduced and can only increase after each changes. On the other hand we have the majority of GIT servers that puts a limit on the maximum file size that they can handle. That is, if you try pushing a file that is larger than a certain size, the GIT push will simply fail and it won't be possible to version that file on this GIT server. The only solution possible on that point would be to increase this max file size limit on the GIT server itself but it requires you to have admin privileges, which git users often don't have. For example, in our case, we're using a github enterprise server that is not managed by us directly. It is impossible to increase that limit. The default max file size limit on the "GitHub Enterprise Edition" is 100MB, you can read this directly on github itself https://docs.github.com/en/enterprise-server@3.7/repositories/working-with-files/managing-large-files/about-large-files-on-github#about-size-limits-on-github-enterprise-server We tried several ways of importing our current project onto our private github repository. All our attempts are failing because of this limit. The last procedure we followed was: https://docs.mendix.com/refguide/on-premises-git/#47-moving-a-subversion-app-to-git There is a good news however since storage of large file size is still made possible through the use of a git extension called GIT LFS. In short, GIT LFS replaces big file size in git by pointers, while storing actual file content on another server. GIT LFS is supported by a large majority of git provider. The only thing a git client must do in order to support it is to download the extension (~10MB) and install it by running "git lfs install". The problem however is that Mendix framework does not support the GIT LFS extension! That is, mendix generates a large .mpr file at the root of every of its project that is doomed to endlessly grow by design and yet mendix does not support the extension that would help circumvent that issue on majority of its GIT server providers. This seems like a nonsense to me. (Or perhaps do you have a plan to break this .mpr file ? ) The question has already been asked before here though: https://community.mendix.com/link/space/studio-pro/questions/114034 This question stays without response now for 1 year and 3 months. As you may know, large organisations have strict security policies that prevent them from hosting their code on a third-party company. Using mendix GIT repositories (configured with a higher max file size) is therefore not an option. On top of that Mendix is sunsetting the support of SVN on its version 10 and will only support versioning with GIT. I contacted the mendix support twice these last months and their reply stays the same: they’re not going to support LFS nor they’re going to break this nonsense .mpr file.    Questions then: - Are you planning to support GIT LFS in a near future? Do you have any roadmap to share with us?→ They’re not going to support it. See answer below: “Git LFS is indeed not supported by Mendix because LFS is designed for the files that don't change often. It is also not expected to be added in the future to our roadmap.” - What are you planning to do to fix the issue of the ever growing .mpr file? Do you have any roadmap to share with us?   - What options do you provide to large organizations wanting to host their mendix project on their own GIT servers?  
asked
6 answers
1

Hello, 

Thank you for your clear answer. 

I agree GIT LFS is not a real solution, it’s only a workaround waiting for the real fix : breaking this MPR.

On my optinion, Mendix cannot say it supports git server providers (Working with Git On-Premises Version Control Server | Mendix Documentation) until it fixes that issue with MPR file because no large project would be able to be versionned on gitlab nor github (max 100MB file size on each)

There seems to be a misalignment somewhere in Mendix: On one hand you will only support git as your VCS on version 10 (Mendix 10 Commits to Git | Mendix), on the other hand, only  project with <100MB mpr file would be able to be versionned on github and gitlab, so exit large projects support. 

It has been more than a year since this issue was first reported: https://forum.mendix.com/link/space/studio-pro/questions/114034 .

On my opinion, this lack of reactivity is concerning (if not frightening) and shows this issue is not a priority at all on Mendix side. 

I hope Mendix will change its priorities soon.

 

 

answered
0

Hi! PM for Version Control here.

 

We are aware of the issues with some 3rd party Git hosts due to file size limitations, such as Github. With the .mpr file growing over time, it’s not possible at this point in time for customers with large .mpr’s to use those services.

 

As mentioned by Support, Git LFS is not a real solution to this problem, as LFS is designed for files that don’t change often.

 

This leaves us with the direction of moving to a different format for Mendix projects. We are investigating what benefits this would bring (enabling Github, possible effects on performance while working with Mendix, ...) and what impact this would have (as lots of parties down the line use the current .mpr format).

At this time I can’t provide clarity yet where we are heading, but I will emphasize this problem is known to us, and we’re investigating how to address it.

answered
0

Thank you for the response. Of the 4 largest Git vendors (Github, Gitlab, Azure, Bitbucket) we only found Github having a filesize limit, which is something we should address in the documentation you're referring to.

We are aware this is frustrating a part of our customer base, and it's clear this needs to be resolved.

Having said that, we are investigating the costs and benefits of splitting the .mpr, and are prioritizing it amongst other version control/Git frustrations.

answered
0

Thank you for your reply. 

For reference, I’m including mendix support answer on a ticket I created that complete your answer above:

---

I have checked your questions and concern with the SDM and PM of the relevant team. It is mentioned that as of now, the only service we are aware of that has a hard limit on file size, which cannot be resolved, is GitHub. We acknowledge the problem caused by this limitation on GitHub.
 
The size of the .mpr file increases as your app grows. One way or another we need to store the model and the metadata associated with that. We have made certain tradeoffs in this regard and are currently exploring whether we need to change our approach by leveraging Git and incorporating planned future features. It is worth noting that the .mpr file format offers several benefits such as consistency and faster reading and writing speeds, which are challenging to replicate with separate files. Nonetheless, we recognize the importance of having a sustainable and performant file format for our metamodel.
 
However, changing the .mpr file format is rather complex. We understand the inconveniences caused by the large .mpr file size and are actively investigating alternatives to move away from a single big binary format (.mpr) to another format. Please note that this investigation is still in its early phase, and even after a decision is made, it will require several quarters of time for implementation.
 
With that being said, we do not consider the size of the .mpr file as a bug. Upgrading Studio Pro will clean up the file, but the overall gains from this process are typically limited and may not sufficiently resolve the issue. While improving the user experience remains a priority, it is not guaranteed yet that we will change the .mpr file format. An example of our commitment to enhancing the user experience is the introduction of automated Git garbage collection, which effectively resolves repository size issues.
 
Until it is resolved, it is not possible to use GitHub with projects that have a large .mpr file. We apologize for any inconvenience caused and appreciate your patience as we work towards finding a solution.
---

answered
0

We have the same issue about MPR size.

 

The MPR size is already over 100MB capacity, and the github policy does not allow this.

 

Did you receive any solutions from the Mendix team to solve this problem?

 

If not, I would appreciate it if you could share how you are solving the MPR size issue in other ways.

answered
0

Bump!

 

Same issue here.  We are currently on version 9, preparing to migrate to version 10.  We have a very large project with an MPR file that is currently ~250MB.  We can't perform the migration to Git in our developer portal because it fails on the file size and number of commits.  We can probably clean up the historical commits to a reasonable number, but the MPR size issue is a show stopper.

 

Suggested solution: It would be a simple matter to split the binary MPR into smaller chunks during the commit process.  Just commit the chunked files, not the full MPR.  When pulling an update, recombine the chunked files into the full MPR locally for normal Studio Pro operations. 

 

I would think this would be a relatively easy pre- and post- action to add to the existing commit and pull process.  People have been chunking large binaries into smaller parts to distribute on services like Usenet for decades.

 

 

answered