It's against professional policy. If you care about him, you should not want him to get in trouble, which could happen if you continue entertaining the idea of being with him and potentially act on your fantasy.
As people have mentioned, teachers often stare at their students. They are bound to, being in a room with them for however long. It is their responsibility to take an interest in and watch over them. And people generally tend to stare when there are other people around. Often, it's unconscious, and unlikely deliberate.
When a person likes another person, it's easy to start to over analyze. Innocent staring can become leering. Simple friendliness can become an unspoken love language. Plain attentive listening can become "oh-em-gee, they are, like, so
And I think friends--I'm definitely guilty--have a habit of feeding ideas they find exciting, because they're fun to talk about. They don't ever think something will materialize from their fun.
If he touches you in any way that's not the equivalent of a high five or terse pat on the back or inquires about your romantic life, I would however be worried for you. And if that's the case, I would advise you to, at the very least, stop seeing him because that is highly inappropriate. It is taking advantage of you.