Why is the worst they can say no?
If you don't ask them, you don't know.
If you ask and they say No, then you know they don't want to date you. That is not the end of the world, this allows you to move on to the next guy.
Either way...YOU need to take it upon yourself to do it. You will never know unless you try.