To make a rectangular selection with rounded corners use the Rounded Rectangle Tool (1)
Select the Paths option and a radius (2).
Optional: For more precise placement use snap to grid or snap to guides (3).
Then turn the path into a selection (4).
This procedure can be used with any of the other shape tools
(Polygon, custom shapes) when the marquee tool doesn't provide the desired shape.

forum discussion: http://forums.dpreview.com/forums/read.asp?forum=1006&message=24345631