Do you think that people who have lived in a neighborhood all their lives move out because they are prejudiced or in fear for their safety and the safety of their children.
Are you suggesting that is the only, or even the main reason?
Populations ebb and flow. Ethnic groups are cconstantly shifting about. Friction is almost always apparent in these times because we tend to feel more comfortable with the familiar.
Here in Vancouver, within the last decade or two, we have seen a huge influx of folks particularly from Hong Kong since the turnover, such that now the central portion of the city and several outlying areas are nearly 50% Asian in background. This has been an enormous change, and not without some kerfuffles, but overall it's been just fine.
Of course rascism goes in two directions, but one's first moral responsibility is to one's own rascism.