How to do this depends on your model structure, but ideally you should be able to use the 'Constrained by' property of your reference selector.
Alternatively, you can use an x-path pointing back to the current object like shown below (it is easier to follow these from right to left, so the example below takes the QuestionTemplate linked to the current object and filters the AnswerTemplate records on the association between the 2).
The disadvantage of using an x-path constraint is that the first association (in this case QuestionTemplate to Current object) needs to be saved before it can be used to filter the answer templates. This is not the case if you can use the 'Constrained by' property
I didn't know about the 'Constraint by' property and it works perfect for my case
Tx a lot