Well, we don't really know what his issue is with you as you haven't said. Are you clumsy? Are you not willing to chat with the customers? Do you not know where everything is? Are you slow? Are you a messy worker?
Perhaps you're just not a good fit or are not trained enough.
Math levels are irrelevant in a customer service kind of job.
And why would you find it 'internally quite funny"? I don't get that.