another gounding question

I recently hooked up a new dryer outlet and connected all the wires, (red, black, white, and ground) to the outlet itself, but did not connect the ground directly to the outlet box. Am I setting myself up for trouble, or will the ground connection to the back of the dryer plug suffice, since it is directly grounded to the circuit panel, which itself is grounded?