Thanks to Marshall and Jasper to help find a solution to my issue.
Below are the steps to clean things up and choose the correct driver:
1) Remove any RedShift .jar files from the /userlib folder
2) Clean Deployment Directory through Modeler (Project->Clean Deployment Directory)
3) Add in this version of the .jar JDBC driver to the /userlib folder:
https://s3.amazonaws.com/redshift-downloads/drivers/RedshiftJDBC42-1.2.1.1001.jar
4) Run app
Cheers