Entering the following command in to your .bashrc seems like a simple way forward.

xmodmap -e "remove lock = Caps_Lock"

However generates the following error when loading multiple terminals

xmodmap:  commandline:0:  bad keysym in remove modifier list 'Caps_Lock', no corresponding keycodes
xmodmap:  1 error encountered, aborting.

This appear to be a safer variant with out the errors when reloading your .bashrc

xmodmap -e "clear lock"