How do you feel? How do you feel about this guy?
If this is all that was bothering me - then I say screw everyone else. It is important how you feel about this guy. Just from hearing this on the surface my guess is people are thinking you are physically more attractive than him. You know what that sort of attraction is short lived. Would you rather have a really hot guy that was a jerk, mean, self absorbed, empty headed?
If you really care for him them that's it - the others opinions do not matter.
If this is the case tell them to mind their own business - you care more than superficial things and your caring goes deeper than skin -- not to mention that everyone has different thoughts and feelings about what is physically attractive.
In regard to potential future medical issues - it is understandable to be worried about him. But you could date anyone and there could be some underlying medical issue you do not know about. Anything could happen to anyone at any time no matter how healthy - you hear about the physically fit high school football player that drops dead at a practice. We are all mortal you just happen to know about his medical weakness and whereas yours is potentially hidden.