@oristarA,
This has nothing to do with insults.
It's a straight choice: Do you agree with (1), (2), or if you don't know, (3).
If you choose (3), you are claiming you don't know enough to say.
Therefore the form of words "you don't know enough to say" makes sense.