Yes, you're free to use HTML5 in any widgets you use, we don't "block" anything. Nothing is preventing you from implementing all of the new HTML5 fancy features in your applications :)
but no, we don't use HTML5 in any of our core tech. This is because a lot of our customers are still stuck with old browsers that don't support it.
In general, HTML5 support does not depend on Mendix but on your browser :) Anyway, several themes use CSS3 features, the Simple Charts widget is based on HTML5 Canvas and in some lost hours i succesfully managed to write a Canvas rendering engine and instruct it using microflows. So, yes you can do HTML 5 stuff.
As far as i know, we do not make use of the HTML5 structural tags yet (which are useful in cases of graceful degredation, but i don't think that is very applicable for our apps yet).