What type of input/widget are you trying to show validation feedback on? Sometimes if it's on a custom widget that an attribute is tied to, it will appear as a popup as that's the behaviour when the validation feedback is unable to find the corresponding input in the client side.
Edit: for using custom js validation to try and mimic the validation feedback action, try something like this snippet from Andries:
// Subscribe to validations of an MxObject
var subscription = mx.data.subscribe({
guid: "123213",
val: true,
callback: function(validations) {
var reason = validations[0].getReasonByAttribute("MyAttribute");
console.log("Reason for validation error on attribute MyAttribute: " + reason);
validations[0].removeAttribute("MyAttribute");
}
});