Trying to do the right thing, but it feels wrong!

Hear me out, bad bits first.

I was sort of having an affair with a man the age of 60, I am 23 and like older men, but this was my first time with an older man.
We never had sex, it was never really about sex, however we would kiss along with a bit of touching but lots of great conversations as we were so alike.
To cut a long story short, it has now ended as I could see a future with him as I fell in love with him, but he being married didn't want to lose everything for me so he's going to stick it out with his wife.

Onto the OK bit.

So, I now know an older man is for me and as I can't have the married man and now that has stopped, I have found myself a man in his 50s who is SINGLE.
My love for the 60 will never completely leave me, so I am trying to put that all in the past and move on.
I have only seen this new man a few times and he has told me plenty of times that he loves sex, me being young etc, I don't have too much sexual experience compared to him obviously.
Today we met for the third time, but already we are kissing, he was touching my boobs etc, although I don't mind, it doesn't feel the same as if the 60 were to do so and although it feels ok, I don't seem to be enjoying it as much as I should.

So although it's hard, I am trying to do the right thing, leave the affair behind and begin with a single man, but one, I can't get the 60 year old out of my head and two, I don't know if I'm rushing things with the 50 year old and is he really going to be obsessed with sex as I'm not really?

My head is a mess, I'm trying to do right but it doesn't feel right, well not at the moment anyways.
I do want an older man, who has sexual experience to teach me etc, so that is what I want, but I can't help but think maybe I'm giving away to much too soon?
Plus the 50 year old said he has had sex with lots of women, whereas the 60 year old who was shy like me, had only slept with a hand full of women as sex wasn't really everything to him.

Any advice on how to move on?
Or am I trying to move on too soon?
Am I being silly and should I just carry on with the 50 year old and go with the flow?

Help needed please!!!