Take the 1869 penny - touted as very rare, but judging by the numbers offered on e bay at any given time, actually not that rare. Nonetheless, always commanding a high price, even in modest grade. By contrast consider some of the die varieties within certain years. Some exceedingly rare, yet somehow never quite capturing the charisma associated with certain years, like the 1933 penny, and the 1905 shilling/halfcrown.
It all boils down to popularity, notoriety and the collective imagination of both the numismatic community and the general public.