I'm aware that you can "edit" binary files with :%!xxd but seriously, am I the only one to think it's totally pathetic? Vim should be better than that.

I use Hexer myself. It's binary editing done right and it should be merged into vim - as in, type :hex or something and the edition mode switches to that editor's layout.