Let's map it out. First off, I am going to assume you are not talking about a brother and sister. You are talking about people from opposite sides of the family, right? Because the truth is, my husband's sister has a son. That boy is nephew to both of us. But you are asking the relation between him and my brother's hypothetical daughter.
The answer is - there is no official relation whatsoever, at least not in English-speaking Western culture, that I am aware of.