You are correct, retrieveXPathQuery only works if the object is persistent and already commited.
The correct way  to retrieve non-persistent or not-commited objects in Java is to use

public static java.util.List<IMendixObject> retrieveByPath(IContext context,
                                                           IMendixObject mxObject,
                                                           java.lang.String path)

https://apidocs.mendix.com/7/runtime/com/mendix/core/Core.html#retrieveByPath-com.mendix.systemwideinterfaces.core.IContext-com.mendix.systemwideinterfaces.core.IMendixObject-java.lang.String

Although the documentation doesn't really state this explicitly. I have set up a small test to ilustrate this:

Here is the code in my Java action. It cretes an Employee and a Project and associates them.

import myfirstmodule.proxies.Employee;
import myfirstmodule.proxies.Project;
import com.mendix.core.Core;



//Create two objects and associate them - do not commit them yet

IMendixObject project_imo = Core.instantiate(getContext(), Project.entityName);
Project project = Project.load(getContext(),  project_imo.getId());
project.setDescription("Very important Mendix Project");


IMendixObject employee_imo = Core.instantiate(getContext(), Employee.entityName);
Employee employee = Employee.load(getContext(), employee_imo.getId());
employee.setName("MyFirstEmployee");

//set the association
employee.setEmployee_Project(project);

//try to retrieve not commited objects
System.out.println("Not commited retrieveXPathQuery: "+ Core.retrieveXPathQuery(getContext(), 
		"//"+Employee.entityName+"["+Employee.MemberNames.Employee_Project.toString()+"='"+project_imo.getId().toLong()+"']"));//[MyFirstModule.Employee[MyFirstModule.Employee_Project='126512316156']

System.out.println("Not commited retrieveByPath: "+ Core.retrieveByPath(getContext(), project_imo, Employee.MemberNames.Employee_Project.toString()));


//commit both objects (synchronously)
System.out.println("Commited project succesfully?: "+(Core.commit(getContext(), project_imo)!=null?"Yes":"No"));
System.out.println("Commited employee succesfully?: "+(Core.commit(getContext(), employee_imo)!=null?"Yes":"No"));


//try to retrieve commited objects
System.out.println("Commited retrieveXPathQuery:"+ Core.retrieveXPathQuery(getContext(), 
		"//"+Employee.entityName+"["+Employee.MemberNames.Employee_Project.toString()+"='"+project_imo.getId().toLong()+"']"));

System.out.println("Commited retrieveByPath: "+ Core.retrieveByPath(getContext(), project_imo, Employee.MemberNames.Employee_Project.toString()));

When we try to retrieve the objects before they are commited the XPathQuery returns empty, but the retrieveByPath correctly returns the object.

After the objects are commited both retrieves find the correct associated object.

Here is the output from the java action call:

Not commited retrieveXPathQuery: []

Not commited retrieveByPath: [[MendixObject:: type:MyFirstModule.Employee id=281474976710957  state:INSTANTIATED]

Commited project succesfully: Yes
Commited employee succesfully: Yes


Commited retrieveXPathQuery:[[MendixObject:: type:MyFirstModule.Employee id=281474976710957 state:NORMAL]

Commited retrieveByPath: [[MendixObject:: type:MyFirstModule.Employee id=281474976710957 state:NORMAL]

I hope this answer helps,

Andrej

There are methods of Core that you can use to retrieve associated objects. Example:
           List<MyAssociatedEntity> myList = new ArrayList<MyAssociatedEntity>();
            
            try {
                List<IMendixObject> vars = Core.retrieveXPathQueryEscaped(context, "//%s[%s='%s']", "module1.EntityA", "someID", someData);
                for (IMendixObject imo : vars) {
                    MyAssociatedEntity mxObj = MyAssociatedEntity.initialize(context, imo);
                    myList.add(mxObj );
                }    
            } catch (CoreException e) {
                logger.error("err", e);
            }