@McTag,
Quote: "Most" usually denotes a significant number.
I would say it may have that connotation, but not that denotation. Literally, most just means something like: "a majority or more."
It's strictly a comparative thing (part to whole). If, out of 3 people, two wear hats, then, with respect to that group it would be appropriate to say "most wear hats," even though its only 2.
But (to respond directly to the post title) "most" cannot also mean "very rare" when talking about the same thing in the same context. Even though it's only 2 in this case, it would be wrong to say, in the case of this group, that it is "rare to wear a hat."