@Kellygirl,
You've got a few different things you need to sort out.
Do you think there is any possibility that there is a future for your marriage - regardless of the affairs both of you have had?
If there is - talk to your husband about what steps he is willing to take to continue in the marriage.
If there is not - contact a lawyer and begin the process needed to end the marriage.
Hopefully your daughter's father will continue to want to parent along with you if the marriage ends. In either case, it is his responsibility to be a good parent to your daughter. Right now it doesn't sound that good. Four year olds aren't meant to live at frat parties. Your daughter's father may need assistance from military counsellors to develop his parenting skills.
Put the affair guy aside for now. You've got a whole life to sort out.
If you've got a strong relationship with affair guy, he will understand this and will still be available to you once you've sorted out your marriage and your parenting arrangements.