Well, that sounds very good and promising ... No wonder you are still after him. I think you just have to make a list about all the good + bad things and see what's the final result.
Imo, if you are emotionally capable of meeting him from time to time without any deeper commitments and forget the relationship part...just go right ahead.
Use him the way you like and he uses you the way he likes. So, both are users then, equally