The upcoming release of Mendix 5 comes with Bootstrap 3, though up to beta-8 Bootstrap 2 is used. Bootstrap 3 is not backwards compatible so Bootstrap 3 themes can not be used with Mendix 5 beta-8 and before and vice versa. That may explain your theme issues.
Beware that as of version 3, Bootstrap encourages the use of a separate theme file, instead of replacing bootstrap.css with a new one. See the default index.html for an example of how a theme should be included.