Since this kind of thing is rather rare, I am going to go out on a limb and ask if you're in treatment for Asperger's disorder. Because from my perspective, if you're incapable of letting go or not bothering someone, that can (at times, not always) be an indication of someone being on the Aspie spectrum.
If you are then this is a talk to be having with the people responsible for your treatment plan. If you aren't, then I would suggest talking to your parents. Most people within your age group aren't required to sit or not sit with anyone - although there may be a concern as you are 4 years older than this other guy. Either way, actually, talk to your parents as this is either, to my mind, a teacher trying to fit things within a treatment plan or you might do well to be evaluated.
Being told who you can sit with at lunch, when you are over the age of 18, is hardly a normal thing to be happening.