Technically, you don't /have/ to. It's a construct of convenience for someone. All one ever really needs to code is a way of doing addition and subtraction, jumping to a particular place in code and some hardware interface.
The code ref allows you to pass around what you'd want to jump to, not all that different from array indexes if you step back a little. Code refs are just values passed around to say what should be executed next. Array indexes are just values that can be recalled to know which value you meant, later.
If they seem nasty, you could always pass around labels you can goto, to. :)