I started kindergarten when I was four yrs old because back then the rule was you had to turn five by the end of the school yr. Now it's different, or course. Anyway, I finished kindergarten, turning five in May. I started grade one and was skipped to grade 2 after three weeks because I could read all the grade one readers. So I was five in grade 2. And I was a very small little five yr old and found that not too many of the grade two-ers wanted to play with me because I was younger and little. Nobody wants to play with 'little' kids when they're young. By the time I got to grade ten, I had figured a lot of it out - nobody really asked how old I was anymore, but I wasn't really into the same things as they were and I was still short. Anyway, it's a pain in the rear - I hear you.
As to your dad - I think just talking to him about the problem might help him become more aware of what you're going through. It's important to communicate these types of things to your parents (in a non-whiny and non-accusatory way) so they can sympathize, try to help, etc.
I don't think going back would help - as someone else said, you'd be bored out of your mind, for one thing, and for another, there's no guarantee you'd be accepted since you're so advanced in so many areas. I think you're unique (except for your twin) in your school and you'll stand out no matter what.
Speaking of your twin, how does she feel? Have you two talked about this?