A flag can not be racist.
It is the people behind it that are.
But a flag, or any other symbol represents ideas groups of people hold.
If you walked around wearing a big sign that said "I HATE _____'s!!!", if really wouldn't be very effective to say, "oh, I don't hate those people, these are just black lines and curves on a piece of cardboard."
Yes, the ink used to make those squiggles is not evil, or racist, but these are the instrument being used to express that hate.
For an important, well known icon I think it is too fine a line to draw between the threads used to make it, and the belief system of some people who display it.
If someone was to paint a pornographic picture of children and animals, it would be a poor argument to make that the paint used to make the picture is innocent.
Some people, like me, simple find the configuration of the colors and lines of that symbol to be racist.