@Pumnellmoo,
Then have more conversations before starting to suggest activities. You may find you don't like him so much. Or maybe more. You're right in that two conversations are really nothing to go on.
So be friendly and say hi when you see him, and strike up short conversations asking how he is, how his weekend was, and what he thinks of something you read recently (news or a book). A few sentences and then you've got to go and do something else.
"Nice chatting with you. See you soon!" Smile and then go.
Do this several times before starting to suggest activities together. Right now, you have a crush on someone who might as well be a stranger.