If you are trying to use the 64 bit version of handbrake on Snow Leopard do not for get that there is currently only one compatible VLC build (Handbrake relies on VLC).

Alternatively use 32 bit Handbrake and the latest VLC