If I understand correctly you now have set the validation in the entity?
I think you will have to do the validation with a custom microflow after an on change action. Because the reference triggers a refresh which triggers the entity validation. And since only 5% of the forms use this validation it is not usefull to set it on entity level. But may be I am understanding you wrong.
Regards,
Ronald