This error usually happens while Jackson is creating JSON. It means a field name was written, but its value was not written, or was written in the wrong order. Because of this, the generated JSON becomes invalid (malformed). Mendix Support also describes this as malformed incoming or outgoing JSON.
1) Check the traffic: enable REST / WebService logs
The first thing to do is to see the request and response payloads of the endpoint. Enable the REST and WebServices log nodes (and Connector if it appears in the stack trace) and check what data is actually being sent or returned.
2) Check the returned data and mappings
For Published REST endpoints, review the returned structure: