Does your theme CSS file define the button differently for a dialog window?
Edit: As mentioned, it depends on the theme you are using. For example, if you are using the standard Dutch theme, opening a form In Content will render the Cancel button with a graphic, but opening the form as a dialog will render it as an Orange background button with no graphic. This is because it uses different parts of the CSS file.
Use Firebug to examine the CSS being applied. When opened in content part of what it uses is
.tundra .mendix_Button_button { -moz-border-radius: 3px 3px 3px 3px; background: url("images/layout/button_bg.png") repeat-x scroll 50% 50% #F6F6F6; border: 1px solid #DDDDDD; padding: 0 6px; }
In a dialog it uses
.tundra .mendixWindow .mendixFormView_controlPane .mendix_Button_button { background: url("images/layout/button_gradient.png") repeat-x scroll center top #C26600; border-color: #AF4900; color: #FFFFFF; padding: 0 5px; }
etc. If you wish to change this you need to develop your own theme, or edit the standard ones. (The standard ones are contained in the deployment.mxz in the Modeler installation)