I think your main problem is lack of an effective communication system in the family. Family communication problems are often the lead cause for many other problems. Modeling excellent communication techniques is a vital role for parents. Healthy communication patterns include both specific skills and a connection with your partner. This includes staying positive, limiting anger and engaging in active listening. When children see parents argue in a healthy way, discuss their days, and actively seek to support one another verbally and non-verbally, they learn how to behave in their own relationships.
There are many things you can implement in your household to improve family communication system. First you and your husband need to be on the same page and for that you probably should discuss with your husband plans on how you want to spend more time with him to improve and strengthen your relationship. You can go out with your husband for walk after dinner for example, or go once to watch a movie or have a dinner outside with your husband only may be once a week. Listen to your husband and share your problem with him just to strength your bond and improve the communication. As a family, start having meals at least three times weekly to promote healthy communication among all family members. Children whose families engage in this important communication-enhancing ritual are more likely to enjoy the additional benefits of improved academic success and greater psychological well-being.
Finally I am not sure which religious group you belong to, but it seems it is important that you and your husband shall practice your religion. It is an important part of our moral values and when we are far from religion, we start compromising ethical values to satisfy our desires.
This is only my assessment based on our conversation and it may not relate to your situation 100% but you are the best judge of your situation. If you think this advise can give you a hope, then I am happy that I could be of any help. If I miss understood anything, I apologize and will like to know more so I can advise you accordingly.