Well, it bothers you enough to come here and ask for opinions.
Frankly, despite being medically impotent, your husband has many other options to satisfy you, had he any intention to do so. Not being affectionate or show his love for you is a definite indication that he has no interest in you other than being the mother of his child.
You only live once, we all don't get a second chance of re-living and if I were in this loveless marriage without affection or love, I'd end it. There are more important things in life than money. Depending on how old your daughter is, your husband will need to pay child support regardless of his threats to withhold money from you.
Everyone has a responsibility in a marriage and if one doesn't uphold their end of the deal, all bets are off.
Good luck to you!