To Katie, The fact that you are wondering is a good thing. Anytime you are uncertain, seeking help is the first step to resolution rather your concerns are valid or not. So give yourself credit for this.
Secondly, sex with a minor under age 16 is a second degree felony. Rape is a felony because it is persuing sex against someones consent. A minor is not (emotionally) capable of giving consent one way or the other because of growth and development, and maturation With this in mind you should ignor these comments given to you about "Did you participate willlingly." A child is a victim and is not capable of participating willingly, and is not at fault for anything in these situations.
Thirdly, it is true that children do silly things. When I was little my (male) cousin and me, who was my age, would play Doctor. It was a simple giggling thing of tickling each other bellies while laying on a table. So your situation may be similar.
Fourth, if you are feeling sincerly uncomfortable with intimacy because this memory keeps popping up in your thoughts when you are with your partner, it is imparitive that you explore this in a way that is comfortable for you. You may consider asking your brother in a way that is not confrontational. There are many good books out there to read up on this subject, which may answer your questions for you. And then, yes, counseling is imparitive.
True sexual abuse for children leaves life long challenges for them that are devistating to their well being, relationship management, and personal happiness.
Fifth, if you truely have been violated you are now on the right track by exploring this question in the first place. Best of luck to you with your quest.