You can take an helper entity, in that entity you can store check box numbers.
You can trigger a microflow/nanoflow on a change event. In microflow/nanoflow you can count the selected check box number and store that number in helper entity.
In the nanoflow make a variable counter. Then for each boolean create a decision and check if the boolean is true. If so add one to the counter. So after each decision you now know how many checkboxes are true.
You can call microflow/nanoflow on selection/un-selection of checkbox in on change vent of checkbox and in that microflow you can put your logic to count the total number checkbox selected and then you can disable or put validation message for user.