Here's a thought:
I have a brother that is almost two years younger than me. We are both in our thirties now, and growing up, we spent a lot of time together.
Sometimes we talk about that time, and it has happened that he told a story about me, only to have me object with "wait a minute, that happened to you, not me". And then we talk it out and sometimes we reach an agreement.
My point is merely that my brother and I shared so many experiences, and so much time has gone by since then, that sometimes memories mislead us. I can have a memory of something happening to me, when it actually happened to my brother, and vice versa. Sometimes our parents tell stories, and mix up which one of us the stories happened to. My point is just that memory isn't always accurate. It is possible that you or your brother remember the story of the purple man because one of you imagined it, and the ocean of time between then and now has altered that impression until it seems it was an experience you both had. Not saying this is how it is, just that this might be what is going on.