@aliyaraya,
You need to clean REALLY well. Just because you cannot smell it, doesn't mean he can't. He most likely can.
Also, walks - lots of 'em. Get that pee outside, and praise him for eliminating outside. Get him to associate going outside with good things. Plus exercise helps with a lot of behavioral issues in general.
Seriously consider crate training, as he'll have a den to go to and they tend to not soil their dens. He can sleep there while you sleep.
Finally, is he neutered? You might want to consider that, if he isn't. I have had neutered dogs and most did not mark, and one only did once - we caught him in the act and yelled no and he stopped. But you need to catch them in the act; even a few seconds later tends to be too late as they just don't make the connection.