For me, rarity could refer to the total number of pieces available, condition rarity, or "rare" eye appeal such as excellent toning. I personally prefer collecting the later two categories. A relatively common coin in rare UNC grade is more exciting to me than a very low grade coin with only a few examples known.
I am not a serious enough collector to have anything particularly valuable but do have a few pieces which think would be hard to upgrade.