Quote: I know what I got myself into, I know he will never leave his wife for me and I don't want him to.
I think you answered your own question there except you don't like the answer.
You don't want him to leave his wife which means he will always put her and his family and certainly himself first. Now you are complaining because you are not first in his life. I don't think you did know what you were getting into.
An affair with a married man might get you some fun times at his convenience, it might get you a lot of nice presents, but it won't make you number one in his life which is what your post implies you want to be.