OM going hot/cold & confused

I'm married, and for last few weeks a colleague who also is married (very recently too) started flirting very strongly over instant messaging at work. We bump into each other in office kitchen, and occasionally talked over IM for work related things. He made it clear before that he found me attractive but I just took that as complement and no more.
Not sure what changed in me but I started reciprocating his approaches and we started very steamy messaging all day long. And every single day it was him who starts the msg going. Couple of weeks ago he started touching me casually - brush on my bottom as we pass each other in corridor when no one is there, which quickly escalated to him getting more physical as we can be at work. I didn't stop him cos I enjoyed the attention. It culminated in us arranging a meeting for work related matter just so we could be in a room privately, and there were lots of physical contact during the meeting, I could see & feel he was turned on; it was mostly touching each other over clothes we were wearing, ending with just couple of kiss. Nothing heavy.
Then all of a sudden, he goes all cold on me. He's the one that has been very strongly pursuing me, he even asked me to find excuse to come into office when he was doing weekend shift just so he can see me. I said no and I didn't go as my husband would have suspected something straight away. So OM was *really* pushing, coming after me strong. But he stops it all of a sudden, as soon as we took things as far as we could.
This has left me feeling rejected. Things have come to abrupt stop because of him and I'm left feeling perplexed - does he want me or what?! If he's going to feel guilty why did he pursue me? He says if we continue as it were it'd have gone much further and he doesn't want that. So why was he pursuing me in first place?
We said we'd go back to how things were before it got more physical, exciting messaging only. But he so far hasn't even done that, it's like he now wants nothing to do with me even though as he says, I've done nothing to offend him. I miss the flirting, the excitement.
I just wish I was the one who put a stop to it. Then I won't have to feel like I've been rejected.