The mendix.css class that is used for tabs within tabs is .dijitTabContainerTabListNested
.tundra .dijitTabContainerTabListNested .dijitTab
= the styling of the tab
.tundra .dijitTabContainerTabListNested .dijitTab .dijitTabContent
= the styling of the tab content
Add .dijitTabChecked to change the styling of the currently active tab.
Adding these classes to your custom css file will change the styling of the tabs application-wide. Use an additional custom class if you only want to change it for specific forms.
You can style specific tabcontainers by adding a custom css class to the tabcontainer. Then you can do something like this:
.yourcustomcssclass .tundra .dijitTabInnerDiv {
border:2px solid #000000;
}
You can easily identify the tab css classes by using firebug. Or search on "tab" in the mendix style sheets: layout.css, mendix.css or tundra.css.