Your StartDate as well as your EndDate are not allowed to be in the date range. And there should be no existing date range where one of the dates is bigger than your start date and smaller than your enddate.
Use an xpath like this:
[(StartDate < $NewStartDate and EndDate > $NewStartDate) or (StartDate < $NewEndDate and EndDate > $NewEndDate) or (StartDate > $NewStartDate and EndDate < $NewEndDate)]
[OrderType = $NewOrderType]
Count the result. If the Count is larger than 0, you have a violation.