@maxdancona,
Culture isn't confined or determined by geography. Geographical boundaries are related to culture but in a more nuanced way than you are indicating.
Also, for you to say one person has no right to speak to another person's cultural authority because of their location, that is a culturally-relative assertion in and of itself. In your culture it might be true, but in mine it might not. My culture might say that people can should speak to others regardless of cultural identity differences or boundaries, and your cultural relativism should respect my POV as such.
Colonialism is relevant in many ways, but it doesn't change right and wrong. If Spanish missionaries brought Christianity to the Americas, for example, and Spanish soldiers killed and infected people, etc. that doesn't mean that Christianity is good for the Spanish and bad for native Americans. Christianity is what it is as a religion separately from what self-proclaimed Christians might have done in its name. Likewise, African Christians are not less Christian or more colonized than white Christians in the world. You have to go beyond narrow, simplistic assumptions about colonialism and culture.