Sorry. I guess I wasn't thinking.
You must clearly decide or you forget about this situation and you continue to live together without jealousy, suspicion, and so on.
Or you always think about it, suspect it, don't trust it and then your family life turns into hell, and most likely you will break up someday.
You can't know if they had a relationship or not.
But if you can no longer trust him, then it may be worth ending this relationship, because your jealousy will start paranoia.