I found a workaround for this. If I associate the chart entity with $currentSession and then set that association when creating the chart entity, the drilldown (by clicking on a datapoint) works every time.
My working theory is that the temporary entities used to support ChartJS are being garbage collected and that associating the chart entity to $currentSession prevents garbage collection of the chart entity and related entities.