As you mention, the correct setting to look at is com.mendix.webui.StateSizeWarningThreshold

To set values in YAML, have you seen the examples located on the git hub (link)?  This value should be set in the same way. 

Just in case you or others are looking for an explanation of this value, that can be found at this link.