@Milly43,
In taking advantage of you, what do you mean exactly?
You've had sex with him already, if you don't want to engage in another sexual encounter with him, then don't! You are the one who says yes or no, you are calling the shots, and don't you forget it !