Is my neighbour being unreasonable?

I live in a small flat with a neighbour above. Our walls are thin. Several times a week after he returns from work, he begins a speakerphone conversation with his family and friends which I can clearly hear in my living room and kitchen. This will usually last from about 4:15pm until around 9:30pm. He conducts most of the conversation by his open window (even in the coldest weather). This makes me feel insecure, invaded and frustrated, as I cannot escape this constantly buzzing sound.
I have asked him courteously to close his window, as this would greatly reduce the transmission of his noise. He refuses, saying that because he doesn't do this late at night, I have no reason to complain. Am I being unreasonable?