Honey, your son will need shoes regardless of how you get along with his father. Plus hanging around with this perpetual manchild never stepping up to do anything about his son is a lousy example for your son. You are teaching him that you can get a woman pregnant and continue to have all the awesome good times without a lick of responsibility.
You need to lay down the law with the baby daddy. This means telling him that he needs to provide support for the child, who is partly his, too. If he refuses parental responsibility and signs away his rights, then you two don't get to hang out anymore. And he may very well take that way out - it certainly sounds to me like he only likes you when you're not pregnant, so what's it to him to even give a second thought to his own child?
If the guy doesn't sign away his parental rights, then he is on the hook for child support. You need an agreement in writing. You need not have a lawyer for this, but that helps immensely as there are probably future financial scenarios that you have not thought of, such as what could happen to your son if you become disabled, or he wants to go to grad school or whatever.
Protect your child's rights and his financial well-being. This is often not about whether he gets to go on a nice vacation but about whether he gets shoes when he needs them. This is all considerably more important than hanging around with this guy.