Just play it off and let it roll off. Your boy-toy sounds controlling and possessive, like he owns you.
You need to learn to be positive. Not a border-line lesbian. Let life be. Nor be some loser who watched "Biker Boyz" too many times and then go "that is my people" bs. Seriously....
Just be upbeat and friendly and let it roll off. Otherwise go into details.
1. Your not really being clear on what exactly happen at all. So far it sounds like nothing happen at all.
2. You claim you and this guy "had a fling" so in reality you and him already have a connection with each other. It is like if I went in for a kiss from a girl I knew that wanted me from a long time ago, and I took her _____ right now. With guys you have to make sure that you state "we are platonic friends, like brother and sister". Where you might be ___ age and still wrestle with each other and get into fights, and dance around among other activities. I was in college and remember this girl who would not STFU about "I have a boyfriend" to another girl who never let me knew she has a boyfriend.
Sometimes you need to just state this fact as many times possible to every new guy you meet. Even if it sounds silly and robotic.
Another factor is to actually be playful, fun, and exciting, and not antagonistically barbaric. It is better then coming off like some kind of messed-up mental border-line-lesbian monster. That could cause problems.
Every-time my family member invest in one of these racist, border-line, karens, fem-nazi, or whatever deranged women I do not spend my money. I just say "you gave ____ money, you spent ____ on _____ " and stress that point with the bills as well. Until the bills get balanced out again.
I would rather root for the next jack-the-ripper, who takes them out then burn my blood on sweat on the next waste of my existence.