Can eating things I’m allergic to cause brain damage?

I am allergic to basically almost every food that isn’t meat or fruits/veggies (even then, some meats too) I am allergic to dairy, wheat, corn, and eggs. But, my symptoms aren’t deadly, just annoying sometimes. However, I recently learned that even with asymptomatic allergies, there is still internal inflammation that affects the body. I also learned that allergies can cause brain inflammation, which can lead to brain damage.

Ever since I was a kid I’ve eaten everything I was allergic to. From 7-8 my mother learned about my allergies and put me on a diet, but I was severely picky (both from just being picky but also coupled with sensory issues that made me dislike many foods even if they didn’t bad to me per se). This caused me to lose a lot of weight, so my mom decided since my allergies weren’t deadly to just let me eat it all.

Now, I’m hooked on sugar and processed foods which are already bad, but I’m also allergic to all of it. And I still struggle with pickiness and texture issues. It’s causing me a lot of anxiety because I’m so scared my lifestyle and consumption of foods I’m allergic to as well as blatantly unhealthy foods has caused permanent brain damage or will. I know the responsible thing to do is basically become a vegan and cut out all my allergies, but it seems very difficult and doesn’t diminish the overwhelming fear that I have towards current brain issues and the possibility of me already having brain damage

So, my question is, is it likely I have brain damage? Do most allergies cause brain inflammation, or brain damage, even if brain inflammation isn’t involved? How should I go about this?