NOTE: A much more current version of this post can be found here.
A GPS receiver discerns your position on the surface of the earth by measuring the length of time it takes to receive signals from satellites. It cannot, however, tell what direction you are facing while standing still (or moving slowly). So if you are navigating to a waypoint, a basic GPS can only point you in the correct direction once you are moving.
This is where an electronic compass comes in handy. It can tell what direction you are facing, even while standing still. Not only does this make navigating easier, it also helps with projecting waypoints and orienting paper maps. Units with an electronic compass usually have a barometric altimeter as well.
Some of the latest models even have a tri-axial compass, meaning you don’t even have to hold it level.
Now for the downside — Any unit with an electronic compass should be re-calibrated anytime you change the battery. Not only that, but they should probably be calibrated again once the charge has dropped significantly. It only takes a minute, but if you are using the compass, you’ll want to be sure to do this.
My Garmin handheld GPS comparison chart shows which type of compass is found on their current models.