You are right, it would also probably reduce server load with a higher user base.

Harassment can go further than just on this site, admins can't do anything outside the media they control.

Public votes are a bad thing, some people are crazy or just assholes.
This will serve as a dis-incentive to participation and engagement.